Operational Concept Validation Strategy Document

FAA/EUROCONTROL Memorandum of Co-operation

Action Plan 5: Validation and Verification Strategies:

Operational Concept Validation Strategy Document

Edition Number Edition Date Status

1

: : :

2.0 27 March 2007 Released Issue

Action Plan 5: Validation and Verification Strategies: Operational Concept Validation Strategy Document

TITLE

Operational Concept Validation Strategy Document Publication Reference: Document Identifier

Edition Number:

OCVSD

Edition Date:

07/03/29-17 2.0 March 2007

Abstract The objective of validation in Air Traffic Management (ATM) modernisation programmes is to provide feed back whether a proposed change will meet the expectations. However the term “validation” is used in many different, even contradicting, interpretations within the ATM community. This FAA/EUROCONTROL Operational Concept Validation Strategy Document (OCVSD) establishes a common understanding of validation in ATM and provides the prerequisite for a better planning, reuse and exchange of validation results of all participating organisations.

Keywords Validation

Verification

Case-Based Approach Scenarios Contact Person(s) Diana Liang email: [email protected] Ulrich Borkenhagen, email: [email protected]

Concept Lifecycle

Structured Planning Framework Concept Validation Unit

Validation Environment Tel +1 202 385 7254 FAA Air Traffic Organization +32 2 729 3318

Publications EUROCONTROL Headquarters 96 Rue de la Fusée B-1130 BRUSSELS Tel: Fax:

+32 (0)2 729 51 51 +32 (0)2 729 99 84

E-mail:

[email protected]

ii

EUROCONTROL DAS/SAS

DOCUMENT APPROVAL The following table identifies all Action Plan 5 participants who have successively approved the present issue of this document.

AUTHORITY

NAME AND SIGNATURE

Ulrich Borkenhagen EUROCONTROL HQ Karen Buondonno FAA Kristina Carr FAA Giancarlo Ferrara ENAV Manuel Dorado AENA Morten Jensen European Commission Sven Kaltenhäuser DLR Diana Liang FAA HQ Nigel Makins EUROCONTROL EC Albert Schwartz FAA Jürgen Teutsch NLR

iii

DATE

DOCUMENT CHANGE RECORD The following table records the complete history of the successive editions of the present document.

EDITION NUMBER

EDITION DATE

INFOCENTRE REFERENCE

1.0

23/11/01

-

1.1

28/05/02

1.2

REASON FOR CHANGE

PAGES AFFECTED

Creation of Document.

all

021001-1

Incorporation of comments.

all

31/01/03

030303-1

Addition of Appendix 1 + editorials

Appendix 1

1.3

08/12/03

031208-01

Addition of Appendix 2 + editorials

Appendix 2

2.0

27/03/07

07/03/29-17

Broad revision to incorporate progress in validation methodology and addition of Appendix 3

iv

all

FAA/EUROCONTROL COOPERATIVE R&D: Action Plan 5 Operational Concept Validation Strategy Document

Contents Page Executive Summary .......................................................................................................1 1. INTRODUCTION.......................................................................................................1 1.1

History and Prospect ....................................................................................1

1.2

Document Structure......................................................................................2

2. OPERATIONAL CONCEPT VALIDATION OVERVIEW.............................................2 2.1

Need for the Operational Concept Validation Strategy..................................3

2.2

Goals of the Operational Concept Validation Strategy ..................................4

2.3

Stakeholders and Requirements...................................................................5

2.4

Planning within Concept Validation ...............................................................6

3. OPERATIONAL CONCEPT VALIDATION STRATEGY.............................................8 3.1

Validation Methodology.................................................................................8

3.1.1 Concept Lifecycle Model .......................................................................8 3.1.2 Structured Planning Framework ..........................................................11 3.1.3 Case-Based Approach ........................................................................13 3.2

Key Performance and Behaviors ................................................................15

3.3

Operational Scenarios ................................................................................16

3.4

Creation of Confidence ...............................................................................19

3.5

Facilitate Effective Collaboration.................................................................21

3.5.1 Validation Information Availability ........................................................21 3.5.2 Validation Environment .......................................................................22 4. CONCLUSION ........................................................................................................23 5. ABBREVIATIONS ...................................................................................................25 6. REFERENCED DOCUMENTS ................................................................................26 APPENDIX A: BEST PRACTICES AND GUIDELINES FOR SCENARIO USE IN CONCEPT VALIDATION .......................................................................................... A-1 APPENDIX B: BEST PRACTICES FOR HUMAN-IN-THE-LOOP EXERCISES.......... B-1 APPENDIX C: SCENARIO USE IN CONCEPT VALIDATION.................................... C-1 APPENDIX D: ICAO’S KEY PERFORMANCE AREAS ............................................. D-1

v


Error! No table of contents entries found.List of Illustrations Page Figure 1. The OCVSD is embedded in the validation process. ........................................4 Figure 2. Planning and decision-making levels interact throughout the lifecycle. .............7 Figure 3. A concept lifecycle model provides structure for validation. ..............................9 Figure 4. The concept selection process may begin with many start-up ideas...............10 Figure 5. The structured planning framework identifies steps within the process...........12 Figure 6. A case-based view provides focused information. ..........................................14 Figure 7. Scenarios have multiple uses within the concept lifecycle. .............................18 Figure 8. Techniques and methods vary during the concept development lifecycle. ......20

vi


Executive Summary Validation is fundamental to the successful research and development (R&D) of operational concepts. Concept Validation is an iterative process by which the fitness-forpurpose of an operational concept being developed is established. Innovative ideas, technologies and working methods cannot evolve from paper concept to operational use without planned validation. The next generation of Air Traffic Management (ATM) concepts involve three key groups: 1. Human system operators, both in the air and on the ground; 2. Engineering partners including R&D teams and industry who will deliver the next generation ATM system; and 3. Organizations, planning and decision-making groups, and individuals who support R&D and make the investment decisions to bring the Next Generation Air Traffic System into existence. This document is written for each of these key groups. They are the fundamental validation partners in the successful development of an operational concept. Although mainly addressed to the Engineering and R&D partners, the other key stakeholders should be familiar with the elements of this strategy document and aware of the key validation concepts proposed. The key validation partners may have radically different perceptions, needs, and outlooks, but they must understand and cooperate with each other for validation to be successful. Together, they address the needs of the end-user of the ATM system - the airspace users. Generally, airspace users are not directly involved in R&D, but their needs are paramount for successful operational concept implementation. Given the global nature of aviation and the similarity of problems and issues facing the United States and European ATM systems, the Federal Aviation Administration and EUROCONTROL have cooperated for many years on different aspects of ATM, including strategy for validating concepts. Therefore, this strategy document applies to all partners in the Memorandum of Cooperation on both sides of the Atlantic. It promotes the development of common goals and principles for validation processes and advocates effective collaboration within and between the United States and Europe. In pursuit of establishing common principle guidelines, the strategy supports a methodology that has three key elements: 1. A concept lifecycle model to help all partner groups have a clear view of where they are in the development lifecycle and to facilitate them in specifying and conducting the appropriate validation for each phase of concept development.

1


2. A structured planning framework that provides a logical structure for planning the validation activities within a program. 3. A case-based approach that develops the idea of conducting systematic analysis on a specific aspect of the concept. Advocating the incremental collection of analyses helps to ensure that the correct results are gathered and recorded in a manner that is comprehensible to all groups. The strategy promotes a more standardized use of operational scenarios as a common language to specify, test, and compare operational concepts. Scenarios are advocated as a means of providing pertinent input to the evaluation being undertaken at a given lifecycle phase of the operational concept development. This implies evaluating performance areas with indicators and metrics and a layered planning approach. The strategy also provides guidelines designed to ensure the credibility of evidence and results achieved during the validation process. Finally, it recommends multiple ways to facilitate effective collaboration including structured methods for effectively integrating, storing, and sharing valuable validation data, and it defines the importance of enhancing our evaluation environments in a common way.

2


1. INTRODUCTION 1.1 History and Prospect In 1997, EUROCONTROL completed the European Air Traffic Management System Operational Concept Document [1] as part of the European Air Traffic Control Harmonization and Integration Program. In 1999, the Federal Aviation Administration (FAA) completed the Operational Concept Validation Process Document [2]. During the production of these documents, the FAA and EUROCONTROL exchanged mutual reviews and experiences in the frame of the Memorandum of Cooperation NAT-I-3454 between the FAA and the EUROCONTROL Organization, Action Plan (AP) 5 Validation and Verification Strategy of Annex 2 (Strategic Air Traffic Management Analysis). In the year 2000, the FAA/EUROCONTROL Research & Development (R&D) Committee requested that the AP5 participants compare those documents and create, if possible, a common validation strategy. The resulting Operational Concept Validation Strategy Document (OCVSD) addressed this request and established a common understanding of the context, purpose, and scope of validation. It presented a set of general principles underlying the approach to be taken. The chairmen of the FAA/EUROCONTROL R&D Committee approved it in June 2002, and it has served as a common point of reference ever since. The AP5 participants identified the need to enhance the strategy with a collection of best practices workshops: •

March 2002: Best Practices for Human-in-the-Loop Exercises (Appendix 1)

•

March 2003: Best Practices in the Development of Simulation Scenarios for Validation Activities in Fast- and Real-Time Simulation (Appendix 2)

•

November 2004: Scenario Use in Concept Validation (Appendix 3)

The AP5 participants held these workshops in collaboration with members of AP9, Air Traffic Modeling of the Operational Concept. They appended the results of these workshops to the OCVSD. In parallel, the United States and Europe gained experience with the application of the OCVSD. In a common effort, the European Commission (EC) and EUROCONTROL developed the European Operational Concept Validation Methodology starting from OCVSD principles. Both the findings of the workshops and the progress in the application of the OCVSD left parts of it outdated. Therefore, in 2005, the AP5 participants decided to update the OCVSD. The newly added material provides an insight into the relationship between concept validation and requirements; a discussion on the role of operational scenarios; and a refined methodology that is traceable, transparent, and accessible. With this update, the

1


authors hope to increase awareness of the Operational Concept Validation Methodology and some of the tools currently in use to support it (e.g., Validation Data Repository, Unified Modeling Language). At the time of publication, both the FAA and EUROCONTROL are projecting their future Air Traffic Management (ATM) systems with global collaboration in mind. The United States is working on their Next Generation Air Traffic System while Europe is developing their ATM Master Plan (Single European Sky ATM Research) program. The OCVSD must continue to evolve to play a valuable role in the collaboration of these efforts.

1.2 Document Structure The remainder of the document layout is as follows: Section 2:

Provides a high-level overview of operational concept validation including the need and goals for a common validation strategy. It also addresses aspects of organizational planning processes and the requirement to understand ATM stakeholders’ needs.

Section 3:

Establishes the key elements of a common validation strategy in support of the defined goals. They include validation methodology, performance areas, operational scenario uses, techniques to create confidence within the community, and tools to promote effective collaboration.

Section 4:

Draws conclusions to support the use of the strategy.

Appendix 1: Provides Best Practices and Guidelines for Scenario Use in Concept Validation. Appendix 2: Provides Best Practices for Human-in-the-Loop Exercises. Appendix 3: Provides Best Practices to Develop Simulation Scenarios for Validation Activities in Fast- and Real-Time Simulation. Appendix 4: Provides examples of Key Performance Areas (KPAs) used in ATM.

2. OPERATIONAL CONCEPT VALIDATION OVERVIEW Validation is often defined as a process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability, and performance. However, this definition is not directly applicable to the earlier phases of concept development. In the context of this document, validation aims to give decision makers sufficient confidence to control and make decisions relative to the development of an operational concept (i.e., fitness-for-purpose). In order to show that the focus of this strategy lies in the early phases of development, we used the term Concept Validation. Concept Validation is an iterative process by which the fitness-for-purpose of an operational concept being developed is established.

2


Fitness-for-purpose covers performance, behaviors, costs, and benefits and can be seen as evidence (often documented in a case-based approach, see section 3.1.3) that demonstrates the appropriateness of the concept examined. It answers the question: Are we building the right system? Concept Validation examines concepts to determine if they are satisfying the objectives of the operational functions and providing operational benefits to stakeholders and thoroughly addressing all aspects. Concept validation within the development lifecycle provides an opportunity to assess each concept to determine its impact on operations prior to significant investment decisions. The complementary mechanism to validation is the process of verification. The relationship between verification and validation during the early phases of concept development is not as well defined as it is in other domains such as software engineering; nonetheless, verification has a constant role in these phases. Verification is the action of demonstrating or proving to be true or legitimate by means of evidence or testimony. Verification activities ensure that the system that is being specified is the one that is being developed. It answers the question: Are we building the system right? This revised edition of the OCVSD provides an approach to deliver the information upon which decisions can be made with confidence. The strategy suggests that validation check points occur at the most important transition milestones during the concept development lifecycle. Validation is not an end test; it is a process applied throughout the development lifecycle. Modernization is based on a system of components, humans, procedures, and interactions that require a sequence of several product development cycles, which will overlap and interact.

2.1 Need for the Operational Concept Validation Strategy The current air navigation systems of the United States and Europe are facing a number of challenges in terms of capacity delivery, effectiveness of operations, and the safety levels necessary to meet demand from the ATM system users. Performance shortfalls that challenge each region are the same, or very similar, in many cases. For example, the regions share the same problems of demand exceeding capacity. Against this background of pressure to improve system performance in the current architecture, there is a need to build and implement new ATM systems that will deliver expected improvements far beyond the capability of today’s systems.

3


To ensure that Air Navigation Service Providers are specifying efficient and costeffective ATM systems, each modernization initiative should be validated before procurement and implementation commence. Due to many influences and causes, several important initiatives to implement change into the ATM system have endured very large time and cost over-runs. Therefore, it is essential to evaluate concepts at early phases of development to ensure that costs and delivery times remain on target and performance of the end-system is as intended. This strategy document expects validation to enable an objective, non-partisan evaluation of any proposed concept. In particular, do not expect validation as a service to substantiate a concept. It is equally possible that an objective evaluation process invalidates candidate concepts. The need to enable more effective collaboration between Europe and the United States in their modernization efforts is underscored by the need for a common approach to concept validation. An effort to apply a shared strategy, encompassing common methods, processes, and terminology will stimulate global harmonization in terms of validation and support the introduction of compatible ATM systems on both sides of the Atlantic.

2.2 Goals of the Operational Concept Validation Strategy There are two main objectives for this OCVSD: to develop common goals and principles and to facilitate effective collaboration between the United States and Europe on concept validation. A shared strategy embedded within the overall validation structure will make it easier for the decision makers of both regions to implement operational concepts in a more timely and effective manner, and it will facilitate the synchronization of the overall master plans (see Figure 1). Validation Exercises by Service Providers

Decision Making Bodies within Regions

R&D Centres

Common Goals and Principles:

OCVSD

U.S. / EC Validation Exercises Figure 1. The OCVSD is embedded in the validation process.

4

U.S. Master Plan

Promoting Interoperable Systems

European Master Plan


2.3 Stakeholders and Requirements The development of operational concepts into successful end-systems becomes a more difficult task as our ATM systems become increasingly more complex. Both in the United States and Europe, there are several key groups (i.e., stakeholders) that carry out the innovative R&D necessary to propose fresh approaches to ATM and to mature those novel ideas into practical solutions. Three key groups involved in delivering next generation concepts include: 1. Human system operators, both in the air and on the ground; 2. Engineering partners including R&D teams and industry who will deliver the next generation ATM system; and, 3. Organizations, planning and decision-making groups and individuals who support R&D and make the investment decisions to develop the next generation system. This document is aimed for these key groups who are the fundamental partners in the validation process. They may have radically different perceptions, needs, and outlooks, but they must understand each other and cooperate for validation to be successful. Together, they in turn address the service needs of the end user. Stakeholder is used to denote all users of the airspace including military air, sea and, land forces, airlines, general aviation, aerial work and recreational aviation. The difficulty for modernization programs is that a crucial element of the development process, the elicitation of requirements, is much more involved and complex than simply gathering information from these stakeholders. In the early phases of concept development, we use the more basic term, “needs,” to express a less stringent application than the term, “requirements.” The problem lies in the fact that in the early phases, these stakeholders express their performance and information needs in many different and often conflicting ways. Understanding, evaluating, and satisfying those needs is not a straightforward process but remains a key element to a successful validation program because it is the stakeholders who will ultimately decide if the concept becomes a reality. The strategy described in the remainder of this document is based on an iterative, model-based approach to extracting and defining all the key ATM stakeholder needs. The idea is that the requirements for a concept evolve over time to determine fitness-forpurpose as part of the validation process. Thus, initial validation activities should focus on identifying candidate stakeholder needs that the end-state concept will ultimately address. We will develop these needs into detailed system requirements later in the process.

5


For example, an aid to help gain agreement on the appropriateness of a concept is to show how it solves specific problems with which the stakeholders can identify. Showing how a concept removes a barrier to improved performance then becomes a key aspect of successful validation. This involves more than just demonstrating how to remove the barriers but also entails describing the overall benefits and the expected costs necessary to achieve the improvement. This process is repeated until the concept is clearly expressed and the key stakeholders are in agreement with the direction of the program.

2.4 Planning within Concept Validation The number of different organizations (including stakeholders) brings creativity and novelty to the development of solutions, and it also imposes an overhead of coordination and cooperation between partners. From high-level aims, we identify a sequence of manageable R&D validation activities. The results of this highly specialized work must maintain traceability to the original highlevel aims to allow senior management to make their decisions. Figure 2 shows an example for the interaction of the different planning levels. The example uses four levels of planning: (1) Strategic, (2) Program, (3) Project, and (4) Exercise. The decision makers, managers, and researchers involved in the process must interact and communicate with each other and coordinate the planning at each level. Planning at the strategic level will incorporate public expectations and has a long-term view. To implement the strategy, programs are established that identify the principal components of the system and place requirements on how these components should develop to meet the strategy. Projects are initiated to develop the components and to pursue how they should operate in the future system. Finally, exercises (e.g., paper studies, experiments, simulations) are planned and carried out to design, refine, and validate the components. An exercise delivers to the project, which delivers to the program, which has the responsibility to deliver the system as set out in the strategy.

6


Planning and Decision Making Levels Lifecycle Phase

Strategic Level Planning Report

Strategic Level (Decision Makers) • Set overall and intermediate goals • Determine timeframe • Determine Key Performance Areas (KPA) • Solve stakeholder issues • Prioritise ATM needs

NO

Goals Achieved ?

Strategy Programme Programme Level Results Report

Programme Level Planning Report

Programme Level (Programme Managers) • Address particular intermediate goal(s) • Focus on particular KPA • Identify interrelations between KPAs • Determine KPIs inherent to KPAs • Define cases necessary to support goals

YES NO

Goals Achieved ?

Project Project Level Results Report

Project Level Planning Report

Exercise Exercise Level Planning Report

Project Level (Project Managers) • Define (or refine) validation strategy • Implement case(s) • Identify hypotheses based on goals/objectives • Produce scenarios and exercise plan • Produce analysis plan

Exercise Level (Exercise Leader) • Prepare platform or facility • Perform pre-exercise testing and training • Conduct validation exercise(s) • Perform analysis • Start reporting cycle on lowest level

YES NO

Goals Achieved ?

Exercise Level Results Report

Figure 2. Planning and decision-making levels interact throughout the lifecycle.

7

YES


3. OPERATIONAL CONCEPT VALIDATION STRATEGY The following section describes a strategy with key elements that are intended to improve the validation of operational concepts. The key elements consist of: •

Validation Methodology

•

Key Performance and Behaviors

•

Operational Scenarios

•

Creation of Confidence

•

Effective Collaboration

This strategy provides guidance in structuring the validation work, and it allows sufficient flexibility to be applied to any operational concept. The approach set forth is not a set of instructions to be followed without question but its application requires expertise to interpret and apply it successfully.

3.1 Validation Methodology This validation methodology includes three aspects of validation that, when viewed together, help provide structure to an iterative and incremental approach to concept development and concept validation. The three aspects are the Concept Lifecycle Model, the Structured Planning Framework, and the Case-Based Approach. The Concept Lifecycle Model facilitates the setting of appropriate validation objectives. The choice of evaluation techniques shows how concept validation interfaces with product development and indicates where requirements should be determined. The Structured Planning Framework facilitates program planning and transparency of the whole process. The Case-Based Approach integrates many evaluation exercise results into key “cases” that address stakeholder issues about ATM performance and behaviors. 3.1.1

Concept Lifecycle Model

Concept validation (evaluating the performance and behaviors of the system described by a concept) and concept development (design and construction of hardware, software, working procedures, scenario development and human/machine interface) are two aspects of the process for changing concepts into reality that cannot be separated. The concept lifecycle model is an attempt to create a structure for the concept validation work that shows the interface to the concept development work. It describes checkpoints in the development of a concept where benefits and costs are examined to avoid continued development when there are no clear benefits.

8


The approach to operational concept development that this concept lifecycle assumes is an incremental and iterative process. Figure 3 shows a lifecycle progression through six phases from the identification of a need to improve ATM performance until the solution addressing the ATM need is implemented. We do not represent the iterative aspect of the process in this diagram. The concept lifecycle described in this document was created based on ideas of other concept development lifecycles; such as the Technology Readiness Levels (TRLs) developed by NASA. This lifecycle is less detailed and less complex than the TRLs to provide a more generic view of the concept development lifecycle. Individual organizations can use this lifecycle and adapt it to their own needs and requirements. Implemented Concept

Idea Focus of OCVSD

ATM needs

V0

Gather and assess ATM performance needs

Scope

V1

Scope operational concept & develop validation strategy & plan(s)

Feasibility

Integration

V2

Iteratively develop & evaluate concept

Pre-operational

V3

Build, consolidate, & test

V4

Industrialisation & Approval

Operational

V5

Implementation

Figure 3. A concept lifecycle model provides structure for validation.

Phase V0 - ATM Needs During Phase V0, a prerequisite to concept validation, the ATM performance needs and barriers must be identified. To complete the validation of the concept, the concept must show that it can remove these barriers, thus enhancing ATM performance. Two key aspects of the eventual validation are: (1) to identify how to analyze the impact of the current barrier and (2) to identify indicators and any associated metrics to evaluate success. These aspects will help address the validation needs identified in this phase. Phase V1 - Scope This is the phase to prepare the initial concept validation strategy and plan. The concept should be described in sufficient detail to enable identification of the potential benefits mechanism (i.e., the change to operational procedures that will enable the known barrier to be removed). As Figure 4 illustrates, the identification of these potential benefits mechanisms can lead to multiple start-ups. Each start-up represents a possible solution leading toward the implementation of the concept.

9


IMPLEMENTED PROCESS

IDEA

V1 V1 V2

V1 V1 V1 V1

V2 V3

V2

V4

V2

V1 V1 Many start-ups

V2 Filtering based on testing of prototypes

V3 use standardised tests and provide full traceability of documentation.

Figure 4. The concept selection process may begin with many start-up ideas.

Because the eventual success of the concept will depend on people other than the developers and evaluators, an important aspect of this phase is the identification of key stakeholders and the establishment of their issues, concerns, and questions that the validation plan must address. The information needs will help to structure the validation plan, identify the types of tests and trials that may need to take place, determine validation analysis needs, and support the establishment of assumptions and constraints about the concept and its potential benefits. Phase V2 – Feasibility Phase V2 is an iterative phase to prototype, develop, and explore the concept until it can be considered operationally feasible. The main focus is on operability and operational acceptance, which entails checking for safety issues. It is during this phase that operational procedures and requirements emerge. The number of iterations depends on the complexity of the concept and how often-unexplained situations occur that need to be explained. Key stakeholders are involved throughout the iterative process to ensure their needs are considered as the concept matures towards an operationally feasible state. At the end of this phase, ensure that the identified performance barrier(s) have been removed. The focus of the next phase is to ensure that the removal of the performance barrier(s) meets the expected system level performance. Phase V3 - Integration This phase integrates the required functionality into prototypes and Fast- and Real-time Simulation platforms to support complex simulations. This phase focuses on integrating the operating procedures by using realistic scenarios that are representative of what the

10


concept must manage in the target end-system. The focus is, therefore, on system level behavior and performance. For example, information about capacity impacts (which can then be translated into various other performance areas such as flexibility, access, and efficiency) will be obtained during this phase. This work will clearly identify costs and benefits and provide information about the potential performance of the overall ATM system. At the end of this phase, it should be possible to show the expected ATM system performance achievement, the cost, and the infrastructure and technologies. Necessary standards for industrialization should also be identified. Phase V4 – Pre-operational Pre-operational preparation takes place during this phase. Prototypes will be transformed into industrial products ready for implementation, and all applicable institutional issues concerning procedures approval will be addressed, such as International Civil Aviation Organization (ICAO) Standards and Recommended Practices and technology standards (e.g., Radio Technical Commission for Aeronautics, European Telecommunications Standards Institute, European Organisation for Civil Aviation Equipment). Additionally, institutional planning of implementation programs will need to be ready for the operational phase. Phase V5- Operational This is the phase when products and procedures are combined to create an operational system. Implementation is a complex and risky procedure, and will require many customized “fixes” to complete the implementation successfully. It is essential that the people who have been involved with the R&D concept maintain a role within the program through the operational phase. The feedback from this phase may become valuable guidance on how to improve the methodology of the earlier R&D phases. The final two phases of the lifecycle model (V4 and V5) are referenced in this document to acknowledge the next steps after the completion of the concept development and the validation. The V4 (Pre-operational) and V5 (Operational) phases are outside the scope of this validation strategy. 3.1.2 Structured Planning Framework This aspect of the common methodology is a guide to planning the validation activities. The operational concept validation process consists of several lifecycle phases (see section 3.1.1), each comprised of many activities from programs through projects to individual exercises. The structured planning framework provides a structure (steps) to those activities (see Figure 5).

11


V1

V0

V2

V3

V4

V5

Exercise Level

Step 0 State/Review Concept and Assumptions Step 1 – Set/Review Validation Strategy Step 2 Determine the Exercise Needs Step 3 – Conduct the Exercise Step 4 – Determine the Results Step 5 – Prepare Information for Dissemination

Figure 5. The structured planning framework identifies steps within the process.

Step 0 would be performed in V0 and would then be revisited at that start of V1, V2 and V3. Step 1 would be performed first in V1 but would be revisited and revised as necessary during V1, V2 and V3. These first two steps cover the validation strategy including planning and identification of projects, studies and supporting cases. Steps 2 to 5 are intended to support the planning of one or more studies on any aspect of the operational concept that requires investigation during V1, V2 or V3. It should be noted that one program could have many studies running in parallel or in series. The steps of the structured planning framework are as follows: Step 0 – State/Review Concept and Assumptions The purpose of this step it is to identify which ATM performance need is being addressed and to determine where problems lie that could prevent achieving that specific need. Measures of the severity of the problem would be established (indicators and metrics) and an initial hypothesis would be made about how the proposed solution resolves the problem. Most of the material in this step should come from sources that exist before a concept validation activity commences. Therefore, much of this activity requires going to those various sources for essential information. This step should be re-examined whenever a new lifecycle phase commences.

12


Step 1 – Set/Review Validation Strategy The activities in this step initiate the process of identifying the stakeholders, issues, aims, objectives, indicators, criteria, and expected outputs of the validation process. All of this information is then compiled into a validation strategy (the strategy is the formulation of the goals or aims of the validation work, i.e. stating which stakeholder questions must be addressed in order to validate the concept). From the validation strategy a validation plan can be created which identifies projects and prioritises which stakeholder questions will be addressed when and how. As the concepts mature through the lifecycle phases the validation plan should be reviewed and updated as necessary or at least at the beginning of each lifecycle phase. Step 2 - Determine the Project Exercise Needs This step consists of selecting the tools and techniques, updating the validation plan and preparing platforms and people for the exercises. This step is repeated as necessary for any exercise or group of exercises during a phase of the lifecycle and should capture the detailed needs appropriate to the phase of maturity. Step 3 - Conduct the Exercise This step is when the exercises are conducted. This step occurs at each phase of the operational concept validation lifecycle. Step 4 - Determine the Results During this step, the exercise results are analyzed and reviewed against the original aims and objectives. This step is continuous throughout the lifecycle. Step 5 - Prepare Information for Dissemination The step includes extracting and processing the information from the exercises into cases for dissemination, such as stakeholder review. This step is continuous throughout the lifecycle. Progression through the structured planning framework may not be linear. Depending on success and any unexpected or unexplained results, it is possible that certain steps will be repeated partially or completely. 3.1.3 Case-Based Approach This aspect of the common methodology is concerned with providing key stakeholders focused information in an easily understood format. The information should be pertinent to the performance barrier that the concept intends to remove as well as the questions to which the stakeholders will need answers concerning the concept and its performance. A case-based approach focused on stakeholder information needs makes it possible to identify what types of studies will need to be performed (see Figure 6).

13


The objective of using a case is to group information into common aspects (e.g., safety, business) in order to describe the potential of the concept under evaluation. This will also support the key stakeholders as they make the investment and implementation decisions. The anticipated principal cases are business, environmental, technology, and safety. Figure 6 illustrates how the cases build up over time (throughout the concept lifecycle) and how they are used to report back to the stakeholders. According to feedback from the stakeholders, validation activities are either continued, redirected, or stopped. If successful, an implementation decision will be made.

Stakeholder performance needs

Stakeholders • ANSP • Investors

Cases

• Operators V0

Barriers to Performance Improvements

• Business • Environmental • Technology • Safety • Others...

• Airspace-Users • Regulators • Technology Suppliers

Information Requirements

V1

Results

V2

V3

• Others...

V4

V5

Figure 6. A case-based view provides focused information.

Business cases manage all information that impact the business aspects of the ATM system and are usually translated into costs and benefits. The KPAs such as capacity, accessibility, flexibility, and efficiency could be built into this case, due to their impact on the financial aspects of the ATM system and its users. Environmental cases capture all the environmental impacts of the concept. Technology cases capture performance requirements, technology assumptions, interoperability issues, etc. Safety cases capture all safety related issues and results.

14

FAA/EUROCONTROL COOPERATIVE R&D: Action Plan 5


Additional cases may need to be added depending on stakeholder needs. For example, there may be a role for a human factors case to group together information relevant to the human role in the eventual ATM system. Additional information about the concept may also be needed to describe where change is happening and what modification will be made to existing operational roles and responsibilities. The cases are a structure for grouping evidence together, taking into account stakeholder information needs. The evidence will focus on ATM performance capabilities, key behavioral characteristics, organizational issues, as well as identification of financial and time-related aspects. The following section discusses how the performance and behavioral aspects are broken down to identify what needs to be evaluated or measured, so that evidence is based on fact rather than on expert opinion.

3.2 Key Performance and Behaviors Measuring the performance of ATM is an issue of growing importance generating much discussion and many papers. Measuring the potential benefits of an operational concept is an important aspect of building cases. The challenge is to find indicators and metrics that allow the stakeholders to understand the impact of a concept on the performance of the ATM system. This means that the performance characteristics of the real ATM system must be decomposed to a level where the measures can be mapped to the system level performance and the evaluated modeling environments. Another challenge is to define key behaviors. Therefore, during the Scope phase (V1) of the lifecycle, it is important to identify two aspects: 1) the indicators and metrics that can be modeled and that will provide a useful representation of the impacts of the concept on system level performance and 2) the key behaviors which the concept must be able to provide. For example, the ICAO has a list of KPAs that provide a framework for describing ATM system performance (Appendix 3). These KPAs are a top-level framework that needs to be decomposed into indicators and metrics. The R&D team will then develop suitable model-based indicators and metrics that are significant to the real ATM system performance indicators and metrics. The decomposition of real-world ATM performance areas into model-based metrics and indicators is the subject of AP2, Air Traffic Operational Concepts. . When evaluating a new concept, the principal KPAs need to be identified along with appropriate indicators and metrics. Once suitable indicators and metrics are identified, it is possible to develop scenarios that will support measurement of the identified indicators. From these evaluation scenarios, the modeling requirements on the validation infrastructure can be identified.

15



In addition to measuring performance, the ability for an end-system to behave in certain ways is also crucial to validation. For example, when Key Behavioral Issues (KBIs) are considered for developing concept validation scenarios, these scenarios should include the ability for an aircraft to make a go-around, to change runways in use, to change destination while in flight, and to maneuver around weather cells, etc.

3.3 Operational Scenarios Operational scenarios serve as a common language for the specification of operational concepts. One of the most important aspects in concept validation is the accurate communication of requirements between the designers, evaluators, developers, and users of the concept. Scenarios are a tool that allows practitioners to describe and test the concept and its intended uses. Carefully developed scenarios can lead to a greater understanding of the concept and can serve as a documented agreement by all stakeholders. Basic elements of a scenario are determined at the highest management level and are usually of a political nature. These elements usually involve defining several stakeholder requirements and provide some of the inputs and outputs to the basic scenario. A short description of potential inputs and outputs follow: •

•

Inputs o

Demand – Current and forecast traffic, unconstrained demand or constrained demand, and demand range

o

Supply – Infrastructure, capacity capability, concept in use description, technical performance

o

ATM Constraints – Environmental, human resources

o

Airspace – Type and characteristics

Outputs o

Performance Key indicators and metrics, KPA’s

o

Behavioral – Operability and usability indicators and metrics for system operators and system users

o

Technical – Description of technical requirements on those enablers

enablers

and

operational

These elements represent some of the basic requirements to a scenario. Once these are known, detailed operational scenarios can then be developed.

16



Scenarios are described by using text or graphics. Appendix 1 contains the scenario guidelines developed at an AP5 practitioner workshop and provides an example of a text-based approach. These guidelines help to develop scenarios by documenting the required information and providing practitioners with a tool to ensure scenario completeness. The guidelines can also be used as an extension of or supplement to the experimental plan of a validation exercise. A completed guideline provides the practitioner and stakeholder with a log of the scenario components and assumptions for the validation exercise. Graphical notation, such as the Unified Modeling Language, can provide an increased understanding of the scenario by depicting specific procedures, actions, actors, phases of flight information, and interaction with tools and systems. This method allows the concept to be communicated graphically with limited text. Using either method provides the necessary documentation for further research on the same concept or for comparison or input in the validation of another related concept. We recommend that a combination of both methods be used. Scenarios are a core component of many types of evaluations and simulation exercises. Their application is not limited to any one type or level of testing. Knowledge of the validation objectives, key issues, validation tests, and scenarios is essential information in establishing the design of simulation exercises and in determining the requirements for non-simulation based assessments. Scenarios also provide the basis for exploratory studies, demonstrations, usability testing, and operational test and evaluation activities. •

Exploratory studies are designed to answer “what-if” questions that provide input into investment decisions. Scenarios used in exploratory studies develop and refine a concept. During these studies, scenarios are developed for both nominal and nonnominal conditions, which provide a complete examination of the concept.

•

Demonstrations are used to show or exhibit a potential capability to a target audience. Scenarios for demonstrations are developed to obtain acceptability from the sponsor or target group. These scenarios usually highlight a certain aspect of the concept, thus allowing the sponsor or target group to see how the concept will perform.

•

Usability testing can help to determine the appropriateness of proposed tools, procedures, and equipment. Scenarios help determine if the human operator can use the system. These scenarios usually consist of a given set of tasks that the human operator must perform to test the system.

•

Operational trials confirm that the system under test performs to predetermined standards and/or criteria when stressed. Scenarios in operational trials are used to determine if the system can operate in the field before implementation.

Inaccurate interpretation of concept requirements or the inability to test the system to its fullest can lead to costly redesign. Therefore, regardless of the type of study or test, scenarios are an important tool in assessing the changes in an ATM system. Scenarios are used throughout the concept lifecycle. Figure 7 depicts the role of scenarios in the different phases of the lifecycle.

17



V1

V2

V3

• Less mature

• Increasing maturity

• Becomes mature

• Requirements loosely defined

• Requirements more clear

• Assessed in rigorous high fidelity experiments

• Several concept options

• Reduced number of concept options

• Used to understand and explore the potential and scope of the concept

• Become more mature

Concept

Scenarios

• Operational concept scenarios • Discriminate between concept options

• Used to establish procedures and phraseology

• Focus on repetition, replication, and performance measurement • Designed with specific indicators and metrics

• Become validation scenarios • Discriminate between remaining concept options

Figure 7. Scenarios have multiple uses within the concept lifecycle.

During V1, the concept is generally less mature with requirements that are loosely defined and open for interpretation. At this phase, scenarios are often used to understand and explore the concept scope. These types of scenarios are commonly referred to as operational concept scenarios. During V2, scenarios become more mature and evolve from an operational concept scenario to a validation scenario. Validation scenarios are designed to set the limits of the concept, establish procedures and phraseology, and determine clear requirements to assist in producing a stable environment for the final pre-implementation phase. These scenarios will assess the impact of proposed changes on the ATM system. During V3, scenarios focus on repetition, replication, and performance measurement. Exercises here will assess a mature concept in rigorous, high fidelity experiments. As such, designers will need to create scenarios with choice of specific indicators and metrics in mind. These scenarios will need to be run repeatedly in order to generate enough data for performance measurements. The outcome of this phase in the validation process will be detailed information about the expected performance of the system under a variety of normal conditions and the ability of the system to cope with various non-normal conditions. Appendices C and D provide the results of practitioner workshops that defined lessons learned in scenario development and the use of scenarios in concept validation during each lifecycle phase.

18



The ability to compare validation results among multiple concepts has led to the discussion of a standardized baseline scenario (see Appendix 1). The use of standard scenarios is an attempt to provide comparability among validation results. Once developed, a standardized baseline scenario can significantly contribute to the validation process and provide valuable evidence required to support a specific concept. A data repository, such as the Validation Data Repository (VDR), would provide a vehicle to host information on standards, such as the standardized baseline scenario, which builds confidence in the validation process through acceptance by multiple organizations.

3.4 Creation of Confidence Concept Validation is a confidence-building activity and can best be carried out if the following elements are implemented during the process. Collaboration Typically, the validation process is dispersed over many organizations or projects. This creates dependencies in terms of information sharing that a common validation process is intended to support. To support knowledge sharing and a uniform view of the process of validation there should be opportunities for practitioners to exchange their experiences and continuously review and maintain the process. Use Proven Methods When validating operational concepts, it is preferable to use established and recognized processes and techniques. Using new techniques, methods, or models for research can often lead to validating the process or technique itself, often at the expense of validating the operational concept. It is recognized that sometimes new techniques or methods are necessary and cannot be avoided. Practitioners should be aware of the increased risks they bring to the timely completion of activities and certainty of results; both can have a negative affect on building confidence. Select the Appropriate Techniques and Methods The application of specific techniques and methods to a lifecycle phase is dependent of the phase of development. In the earlier phases, they are often less detailed, less expensive, and more flexible. In later phases, the level of necessary fidelity tends to increase (see Figure 8). Practitioners should be aware of the strengths and limitations of the different techniques and methods used in validation. They should apply them appropriately and ensure the stakeholder clearly understands the limitations at the beginning of the process.

19



Figure 8. Techniques and methods vary during the concept development lifecycle.

Plan with Flexibility Plans must be prepared at the beginning of the lifecycle when information is at the minimum. Assumptions must be made for the missing information. These assumptions must be explicit and reviewed often. As a result, validation activities should be planned as much as possible; not only the engineering aspects, but also the total project management portfolio. A structured and iterative approach to planning should be adopted (see section 2.4 for details of Planning within Concept Validation). Demonstrate Competency Knowledge about the competency of the organization carrying out the validation generates confidence. This is also a potential source of both real and false confidence. One of the means to discard false confidence is to adopt established standards from other related industries, such as the International Organization for Standardization (ISO) or the European Foundation of Quality Management. Avoid Inadequate or Incomplete Evidence Experience shows that confidence built on inadequate or incomplete evidence is false hope. All available evidence collected during the validation process should be made available, even when results are contradictory, unexpected, or unexplained. Present Results Unambiguously Practitioners should not confuse precision with accuracy when producing and presenting results. They should use the same scale of axes when presenting similar results to avoid misleading interpretations. Conclusions made from validation results should be

20



clear and concise and should always include some measure of the uncertainty of the result. Involve the Stakeholder(s) Practitioners should engage key stakeholders throughout the lifecycle process, particularly during requirements development. When key stakeholders express requirements clearly and logically, practitioners have an expectation of delivering good results in which all have confidence.

3.5 Facilitate Effective Collaboration Effective collaboration between the FAA and EUROCONTROL is a key objective of the validation strategy. One way to facilitate collaboration is by promoting mutual access to valuable validation information. In addition, improved interoperability and integration of validation tools will support this goal. Applied together, these will dramatically improve the speed at which validation activities progress as well as improve the decision-making process with concept lifecycles. 3.5.1 Validation Information Availability The strategy put forth implies a structured and systematic method for keeping validation information available, which, if applied, promotes mutual understanding and integration of the information. Having access to validation information should •

facilitate and ease the communication between previous and ongoing ATM concept validation programs, projects, and exercises; and

•

involve various user groups/stakeholders in the validation process by providing them with easy and timely access to consolidated and current validation data.

The content of the captured information should allow for straightforward analysis of validation data. At a minimum, it should include •

an overview of the high-level validation objectives in terms of the concepts developed, the scenarios investigated, and the benefit targets required;

•

an ability to structure the detailed validation objectives of the individual projects and monitor how they are achieved;

•

a view of past, current, and future validation work that enables the evidence for assertions derived from the work in the context of the established objectives (i.e., traceability); and

•

a view of how the diverse validation activities are performed (such as identifying the validation tools, techniques, standard scenarios, etc.) to produce results and associated findings that address the objectives.

21



One example in line with this strategy goal is the EUROCONTROL operated VDR, which provides access to validation information. User requirements for the VDR were solicited in a process open to all stakeholders and with participation from the FAA. There is joint commitment by the EC and EUROCONTROL to use the VDR to house their ATM validation results. The FAA has commenced data input for the VDR as well. The VDR is publicly available, free of charge, via the Internet [3]. A VDR team administers and maintains the repository in close cooperation with its user community. The team supports all users operating the VDR, oversees the input of information, and ensures its adequate quality. Although the content of the VDR is constantly growing and already covering many areas of ATM validation, it is not intended to be a single source of R&D-related validation information. For example, the VDR is complementary to other sources such as the Analysis of Research & Development in EUROCONTROL Programs [4]. The intent is to promote the use of such tools and to disseminate valuable validation information within the research communities. 3.5.2 Validation Environment The validation environment describes the infrastructure required for the validation activities throughout the concept development lifecycle. It may also provide a framework for eventual implementation of the operational concept. The ultimate goals of the validation environment are to achieve interoperability and integration of tools and to promote the re-use and exchange of data and results. A technically mature validation environment capable of meeting performance requirements (e.g., response times, quality of information, Human Machine Interface) is important because it may have a strong impact on the results of validation activities (e.g., simulations). The design, collaboration, and interoperability features of the validation environment point to a system architecture in which all participating organizations can easily integrate platforms, tools, and support systems. Such an open, flexible, modular validation environment should facilitate the • • • • • •

addition of new components, replacement of components, exchange of data, common data management principles, dissemination of the validation environment to different locations, and use of the validation environment by many users.

22



Among the components of the validation environment, good data preparation tools and practices allow data exchange between Fast- and Real-time Simulation. For example, good data exchange between fast-time and real-time simulators will permit rapid calibration of a ”controller model” based on real-time recordings. In addition to the reuse of prepared data, the validation environment should also facilitate the exchange of other data types (e.g., radar data, flight plan data). Developing and using validation environments in a common way is expected to provide relevant cost and time savings together with an improved quality of validation results.

4. CONCLUSION An organized and structured approach to validating operational concepts is essential for researching and developing the next generation ATM system. In this operational concept strategy document, we have set forth a structured approach to concept validation. We have proposed a concept lifecycle view covering the evolution of operational concepts from the idea phase through to service implementation. The concept lifecycle helps those involved in validation to understand at what phase of the lifecycle the validation is being performed and supports the tailoring of the validation activities to the actual phase of the lifecycle. This validation strategy document focuses mainly on the V1 to V3 phases of the lifecycle, however, concept validation is really a continuous activity carried out hand-in-hand with concept development throughout the entire lifecycle. In addition to the concept lifecycle, a structured planning framework is advocated for the planning of individual validation activities or experiments. This approach is targeted at providing a repeatable process which ensures transparency of the validation work to stakeholders. Identifying and working with stakeholders in a structured fashion helps to increase the potential for successful validation. As a third element, the strategy promotes the use of a case-based approach to validation as well as collecting and collating results. A case-based approach collates key information in a form accessible to those who need it. A case-based approach will drive requirements for producing specific results and supports the incremental development of validation results over the lifecycle. The strategy also provides a shared definition of validation and articulates the relationship between stakeholders and requirements. It sets out the infrastructure necessary to support cooperative and collaborative validation and, finally, it highlights the overall purpose of validation to provide confidence in our future concepts.

23



Verification is frequently mentioned when talking about validation; indeed, verification and validation is common when developing concepts and testing processes. It is intended that when the current edition of the concept strategy is accepted, the next edition will address the verification issues that arise when developing new operational concepts. This document represents continued cooperation between the United States and Europe on concept validation. With the “ground rules” in place, the collaboration will continue on a sound foundation. The body of work associated with concept validation has expanded and will continue to develop in the future.

24



5. ABBREVIATIONS AP AP 2 AP 5 AP 9 ATM EC FAA ICAO ISO KPA KBI OCVSD R&D TRL VDR

Action Plan Action Plan 2, Air Traffic Operational Concepts Action Plan 5, Validation and Verification Strategies Action Plan 9, Air Traffic Modelling of Operational Concepts Air Traffic Management European Commission Federal Aviation Administration International Civil Aviation Organization International Organization for Standardization Key Performance Area Key Behavioral Issue Operational Concept Validation Strategy Document Research and Development Technology Readiness Level Validation Data Repository

25



6. REFERENCED DOCUMENTS [1]

European Air Traffic Management System Operational Concept Document (OCD), European Air Traffic Control Harmonisation and Integration Program Doc: FCO.ET1.ST07.DEL01, 1 March 1997.

[2]

Operational Concept Validation Process, FAA September 1999, prepared by NAS Advanced Concept Branch, ACT-540.

[3]

Validation Data Repository, www.eurocontrol.int/eatmp/vdr

[4]

Analysis of R&D in EUROCONTOL’s Programmes, http://www.eurocontrol.int/ardep

26


Appendix A Best Practices and Guidelines for Scenario Use in Concept Validation

Prepared By: Andrew Harvey EUROCONTROL Experimental Centre Karen Buondonno, Parimal Kopardekar, Sherri Magyarits Federal Aviation Administration William J. Hughes Technical Center Nicole Racine Titan Systems

3 February, 2003

1

Acknowledgments The persons listed below attended the March 2002 workshop in Atlantic City, New Jersey. This paper is a collection of their ideas, experience, and expertise. The authors would like to thank them for their dedicated involvement and contributions to this effort. BAE Systems: Virginia Saulnier CENA: Didier Pavet Deep Blue and University of Sienna: Patrizia Marti EUROCONTROL Experimental Centre: Robin Deransy, Alistair Jackson, Barry Kirwan, Nigel Makins ENAV: Giancarlo Ferrara EUROCONTROL: Ulrich Borkenhagen, Hans Wagemans FAA Civil Aerospace Medical Institute: Carol Manning FAA Headquarters: Diana Liang, Jacqueline Rehmann FAA William J. Hughes Technical Center: Richard Ozmore, Ben Willems, Tanya Yuditsky ISDEFE: Nicolas Suarez MITRE CAASD: Celesta Ball, Urmila Hiremath NASA Ames Research Center: Sandy Lozito NASA Langley Research Center: Kara Latorella, Dave Wing NLR: Michel de Bruin, Juergen Teutsch San Jose State University: Lynne Martin University of Siena: Antonio Rizzo Volpe Transportation Research Center: Kim Cardosi

A-2

Table of Contents 1 Background ...............................................................................................................6 1.1

Purpose of the Appendix...............................................................................6

1.2

First United States/Europe Workshop on Best Practices for Human-in-theLoop Exercises .............................................................................................6

2 Air Traffic Management Initiatives and Validation ......................................................7 2.1

Impact on the Human Operator.....................................................................7

2.2

Overview of Validation Techniques...............................................................9

2.3

Choosing a Validation Technique ...............................................................11

2.4

Choosing a Simulator and Assessing Fidelity .............................................12

3 Best Practices for HITL Exercises ...........................................................................14 4 Best Practices for Before the HITL Exercise............................................................15 4.1

Managing the Process ................................................................................16

4.2

Experimental Design Considerations ..........................................................18

4.3

Airspace and Scenario Development Considerations .................................21

5 Best Practices for During the HITL Exercise............................................................24 5.1


5.2

Experimental Design Considerations ..........................................................25

5.3

Subjective and Objective Data Collection....................................................26

5.4

Validation Data Storage ..............................................................................27

6 Best Practices for After the HITL Exercise...............................................................28 6.1


6.2

Statistical and Operational Significance of Results .....................................29

7 Summary of the HITL Exercise Development Process ............................................30 8 Overall Summary.....................................................................................................31 References ..................................................................................................................33

A-3

List of Tables and Figures Table 1: Human-in-the-Loop Factors............................................................................... 8 Table 2. Validation Steps and Corresponding HITL Techniques.................................... 12 Table 3. Example of an Importance-Performance Matrix............................................... 14 Table 4. Best Practices for Before the HITL Exercise ................................................... 15 Table 5. Best Practices for During the HITL Exercise ................................................... 24 Table 6. Best Practices for After the HITL Exercise ...................................................... 28 Figure 1. Sample Validation Route Map (Banana Model) .............................................. 11

A-4

Executive Summary The first Federal Aviation Administration (FAA)/EUROCONTROL Action Plan 5 workshop was conducted on March 19-21, 2002 to discuss and develop best practices related to human-in-the-loop (HITL) exercises. The workshop was well attended by 14 European participants and 19 United States (U.S.) participants. The participants were all practitioners experienced in validation research and HITL exercises. The original intent of the workshop was to focus only on identifying best practices for real-time HITL simulations, however, during the course of the workshop, the practitioners agreed that many of the identified best practices could also apply to a variety of HITL exercises, not just real-time HITL simulation. The following topics were discussed: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Role of HITL exercises in the validation process, General overview of HITL simulation process, Managing HITL validation exercises, Scope and fidelity considerations, Experimental design considerations, Airspace and scenario characteristics, Subjective and objective data collection, Sources of error and variance, and Statistical and operational significance of results.

Each topic was introduced by a short briefing and followed by a discussion session facilitated by a moderator. The participating practitioners performed all briefings and moderation. After the initial discussions on each topic, the participants were divided into working groups to further discuss and identify elements of the best practices for the topic assigned to them. Their consolidated recommendations were presented to all participants for final discussion and consensus. All discussions were recorded. The recommended best practices described in this appendix are the results of collective input from all of the European and U.S. participants. The intention is that these best practices serve as supplemental guidelines for experienced practitioners who perform HITL validation activities. Though different from the order which they were discussed in the workshop, the best practices are presented in harmony with the five steps outlined in the High Level Methodology Approach defined previously in the main body of this document. The best practices from each of the topics are organized by the periods before, during, and after an HITL validation activity. Overall, the participants felt that the workshop was very valuable and useful for their research endeavors. They also suggested that the further workshops should be arranged to discuss additional topics such as statistical analysis, metrics, fast-time simulations, modeling, safety assessments, and advanced concepts.

A-5

1. BACKGROUND The Federal Aviation Administration (FAA) and EUROCONTROL agreed upon an Action Plan 5 (AP5) in November 1997. The goal of AP5 is to determine a strategy for validating and verifying the performance, reliability, and safety of Air Traffic Management (ATM) systems. Both the FAA and EUROCONTROL had developed separate validation strategies. Recently the organizations have worked to blend the documents into a harmonized strategy plan for validation and verification that can be adopted universally and serve as a best practices guidance to research practitioners (see document that precedes this appendix). By adopting such best practices, the validation process can be improved while being accomplished more timely and more cost-effectively. The application of the recommended strategies and practices may sometimes require unique but minor aberrations based on cultural, fiscal, and organizational needs.

1.1

Purpose of the Appendix

In addition to the overall strategy guideline, one of the objectives of the FAA/EUROCONTROL AP5 was to develop detailed best practices for all aspects of the validation process. Validation exercises take many forms including: analytic studies, concept studies, fast-time simulations, part-task and full-mission real-time human-in-theloop (HITL) simulations, field-testing, and shadow-mode testing. Development and subsequent use of these best practices will allow for sharing of information and comparison of results. Best practices covering the use of metrics, data collection, data analysis, and reporting are needed for each type of validation exercise. These best practices take into account several factors including resources, cost, and most importantly, prior lessons learned. In order to capture lessons learned, a series of workshops will be organized. The attendees of these workshops will be experienced practitioners in their respective fields. The purpose of this document is to describe the best practices related to HITL exercises for validation of ATM initiatives.

1.2

First United States/Europe Workshop on Best Practices for Human-inthe-Loop Exercises

The FAA and EUROCONTROL organized the first AP5 workshop in March 2002. The objective of this first workshop was to discuss and identify best practices from practitioner experience with HITL exercises. There were 19 practitioners from the United States (U.S.) and 14 practitioners from Europe. The U.S. participants included researchers from the FAA William J. Hughes Technical Center, FAA Civil Aerospace Medical Institute (CAMI), National Aeronautics and Space Administration (NASA) Ames Research Center, NASA Langley Research Center, Volpe Transportation Research Center, and MITRE. The participants from Europe included researcher practitioners from EUROCONTROL, EUROCONTROL Experimental Centre, Deep Blue, University of Sienna, Ente Nazionale di Assistenza al Volo (ENAV), Centre d'Etudes de la Navigation Aérienne (CENA), Ingeniería de Sistemas para la Defensa de España (ISDEFE), and Nationaal Lucht en Ruimtevaartlaboratorium (NLR). The results of the workshop are documented in this appendix.

A-6

2. AIR TRAFFIC MANAGEMENT INITIATIVES AND VALIDATION Sponsors, managers, and researchers often face the difficulty of determining needed validation exercises for an ATM modernization program. The scope and level of validation vary according to the type of the program. Generally, the modernization program that aims to achieve a positive change can be categorized as: • • • • •

2.1

Development of decision support tools (e.g., Traffic Management Advisor, Medium Term Conflict Detection), Procedural changes (e.g., Reduced Vertical Separation Minima), Advanced concepts (e.g., Dynamic Resectorization), New software/hardware (e.g., Standard Terminal Automation Replacement System, Display System Replacement, Medium Term Conflict Detection), and Advanced technology (e.g., Global Positioning System, Data Link).

Impact on the Human Operator

Validation activities vary for different types of programs. The level and importance of validation efforts also vary. They depend upon the potential changes that a modernization initiative may produce. The impact of an ATM initiative on the human operator is of primary importance and is therefore a key element of the validation process. Table 1 describes a sample of HITL impact factors potentially generated by a modernization program.

A-7

Table 1: Human-in-the-Loop Factors The following is a list of factors that could be impacted by an ATM modernization program (FAA Human Factors Job Aid, 1999): 1. Workload: Operator and maintainer task performance and workload. 2. Training: Minimized need for operator and maintainer training. 3. Functional Design: Equipment design for simplicity, consistency with the desired human-system interface functions, and compatibility with the expected operation and maintenance concepts. 4. CHI/HMI: Standardization of computer-human interface (CHI)/human-machine interface (HMI) to address common functions, employ similar user dialogues, interfaces, and procedures. 5. Staffing: Accommodation of constraints and opportunities on staffing levels and organizational structures. 6. Safety and Health: Prevention of operator and maintainer exposure to safety and health hazards. 7. Special Skills and Tools: Considerations to minimize the need for special or unique operator or maintainer skills, abilities, tools, or characteristics. 8. Work Space: Adequacy of work space for personnel and their tools and equipment, and sufficient space for the movements and actions they perform during operational and maintenance tasks under normal, adverse, and emergency conditions. 9. Displays and Controls: Design of displays and controls (to be consistent with the operator’s and maintainer’s natural sequence of operational actions). 10. Information Requirements: Availability of information needed by the operator and maintainer for a specific task when it is needed and in the appropriate sequence. 11. Display Presentation: Ability of labels, symbols, colors, terms, acronyms, abbreviations, formats, and data fields to be consistent across the display sets, and enhance operator and maintainer performance. 12. Visual/Aural Alerts: Design of visual and auditory alerts (including error messages) to invoke the necessary operator and maintainer response. 13. I/O Devices: Capability of input and output devices and methods for performing the task quickly and accurately, especially critical tasks. 14. Communications: System design considerations to enhance required user communications and teamwork. 15. Procedures: Design of operation and maintenance procedures for simplicity and consistency with the desired human-system interface functions. 16. Anthropometrics: System design accommodation of personnel (e.g., from the 5th through 95th percentile levels of the human physical characteristics) represented in the user population. 17. Documentation: Preparation of user documentation and technical manuals (including any electronic HELP functions) in a suitable format of information presentation, at the appropriate reading level, and with the required degree of technical sophistication and clarity. 18. Environment: Accommodation of environmental factors (including extremes) to which it will be subjected and their effects on human-system performance.

A-8

2.2 Overview of Validation Techniques Typically, the validation process involves multiple methods, techniques, and tools. The scope and resources needed vary depending on the level of maturity and type of ATM initiative that is being validated. The following techniques are typically employed for validation exercises: 1. Concept studies/Paper studies/Analytical studies - Concept studies are performed at particularly early stages of an ATM initiative. These studies address technological feasibility, engineering analysis, benefits (or hypothesis), and analysis of a concept or an initiative. Paper studies are used for conducting a target level of safety analysis, risk analysis, cost-benefit trade-off analysis; and theoretical examination of aspects of the concept of operations. These studies may include gap analysis, functional decomposition, comparative studies, analytical studies, etc. 2. Task analysis - A task analysis focuses on identifying detailed tasks and subtasks. Such a detailed breakdown is useful for determining the division of activities within a team (e.g., executive controller and planner controller or Radar Controller and Radar-Associate). The breakdown is also useful for identifying any routine activities that are potential candidates for automation. 3. Storyboarding - The storyboarding technique is primarily used in early stages of a concept or ATM initiative. This technique is typically performed by a team consisting of subject matter experts (SMEs), human factors researchers, engineers, and other necessary members. This technique is particularly useful for examination of concepts and considerations such as information flow, procedural needs, and decision support tool functionality. Typically, the storyboarding involves drawing sketches and pictures (as if a story about a concept is being described with pictures, hence the name storyboarding) that are used for describing the concept or used as talking points to generate discussion about the concepts. 4. Cognitive Walkthroughs - Cognitive walkthroughs are also used in the early stages of concept exploration. Cognitive walkthroughs are used to discuss issues such as human error potential, data/information flow between operators and between operators and machines, information needs and decision support tool functionality, and procedural considerations. Storyboarding, task analysis, data flow diagrams are some of the specific techniques that are used during the cognitive walkthrough process. Typically, cognitive walkthroughs are performed by a team of SMEs, human factors researchers, engineers, and other members as appropriate. The walkthroughs provide a structure for team members to mentally imagine the concept and walk through the details of the ATM concept (hence the term cognitive walkthrough). The cognitive walkthroughs are useful during design reviews and identification of potential issues with integration of multiple decision support tools and procedures. 5. Fast-time simulation/modeling – This technique is based on human models but no human interaction is employed. All scenarios are compiled via computer-based simulation. Fast-time simulation and modeling exercises are typically performed to examine system performance including benefits assessment (e.g., delay, fuel burn, time/distance flown), and the analysis of capacity, safety, risk and efficiency. They are often used in the early stages of validation to get initial preliminary ideas of potential benefits. Fast-time and modeling studies are also useful for identifying

A-9

potential problem areas where real-time simulation studies are necessary for further exploration. 6. Rapid Prototyping - Rapid prototyping studies provide an opportunity to develop HMIs for advanced concepts and conduct usability studies. Although rapid prototyping can be considered a development activity, the user-interface is often tied with the human and system performance. In some cases, rapid prototyping exercises will provide input to real-time HITL simulation studies and vice-versa. Prototyping is typically used to demonstrate the look and feel of an interface that will support a new technology or a concept. Increasingly, iterative prototypes are being used to identify requirements for user-interfaces. During the iterative prototyping process, SMEs are provided with initial prototypes of interfaces and based on their input the prototypes are modified. This process is used to generate team consensus on user-interface requirements. 7. Surveys and interviews - Surveys and/or interviews of SMEs are performed to gather their perspective on new initiatives. They focus on their opinions about feasibility, benefits, and acceptance. Such data can be useful, although not sufficient alone, to modify a concept. 8. Real-Time HITL Exercises 8.1. Part-task Real-Time HITL studies - Part-task real-time HITL studies are performed to examine a specific topic or question. For example, a part-task, real-time HITL study may focus on issues particularly related to ground-side issues involving only air traffic controllers, or it may address air-side issues involving only flight crews. They are generally performed to identify and assess specific human performance issues as a result of new ATM initiatives (e.g., impact of new data link technology on communications duration). 8.2. Full-mission Real-Time HITL studies - Full-mission real-time HITL studies are performed to examine or demonstrate an end-to-end concept. For example, a full-mission real-time HITL study will likely include cockpit simulators, air traffic control simulators, flight crews, and air traffic controllers. The scope and fullmission varies based on the objectives of the particular study or demonstration. For example, a study may include simulation of entire day’s ATM activities for a particular airport to assess capacity increases based on the addition of a new runway. A different kind of full-mission study may focus on a specific concept such as “shared separation” and its impact on the flight crew and air traffic controllers simulated in generic airspace. 8.3. Shadow-mode testing - Shadow-mode testing is particularly useful for new hardware and software initiatives. In this technique, an operational prototype is fed live data but is not used to control live events. This technique involves the simultaneous operation of both old and new systems in the operational setting. Such side-by-side testing allows for a ‘real-world’ evaluation of stability, reliability, performance, and acceptability of a new technology while the old technology is still operational, hence the term shadow-mode. 8.4. Operational Trials – In this technique and operational prototype is fed live data and is used to control live events. Operational trials are performed to demonstrate the feasibility and benefits of advanced concepts such as new technology, procedures, or decision support tools. These trials are usually performed in the later stages of the validation process. Initially, operational trials are performed under nominal scenarios such as good weather, low traffic

A-10

volume, etc. to ensure the feasibility of the technology (e.g., data link, ADS-B). As the initial trials become successful, the operational trials are further performed under increasingly complex situations. It should be noted that not all of the activities previously described are considered HITL techniques, however all of them to some extent rely on empirically derived human models (e.g. fast-time simulation). Within an ATM modernization program, validation exercises should be carefully organized as part of an overall validation strategy plan. Figure 1 depicts a sample validation route map (commonly known as the Banana Model) that illustrates how various validation activities could be linked together. It must be noted that the different validation activities can be performed sequentially, in parallel, or iteratively depending on the need and scope of validation exercises, and you don’t necessarily have to use all of them. Analytic modelling

Fast-time simulation

Field test

Small-scale real-time simulation

Shadow-mode trial Operational trial Operations

Large-scale real-time simulation

Figure 1. Sample Validation Route Map (Banana Model).

2.3

Choosing a Validation Technique

As previously stated, validation is an iterative and incremental activity. Advanced ATM concepts often require complex understanding of operations and their implications on procedures, decision support tools, and human factors. It is unlikely that one exercise will answer all the validation questions. Therefore, the roadmap to validation should comprise a series of validation activities, many of which will be involve HITL studies. It is recognized that the maturity of an advanced concept is one of the factors in deciding the type of technique that is suitable for the validation. There are a number of ways to describe the maturity of a concept. For example, NASA uses Technology Readiness Levels (TRLs). Another method is to use the five basic validation steps proposed by

A-11

FAA/EUROCONTROL Action Plan 5 to describe the maturity of a concept. Table 2 summarizes the validation steps (labeled V1 through V5) and suggests different validation techniques that can be used at different phases of the process. It must be recognized that these steps are very generic and need to be interpreted according to the concepts and individual agency’s processes. The validation steps are defined in more detail in Section 5.2 of the main body of this document. Table 2. Validation Steps and Corresponding HITL Techniques

Validation Steps

HITL Technique

V1- Basic principles of new concept

Interviews, surveys, cognitive walkthroughs, data flow diagrams, task analysis, storyboarding, and analytical studies

V2- Initial proof of concept

Cognitive walkthroughs, functional decompositions, storyboarding, analytical studies, fast-time modeling, demonstrations, prototyping, and part-task HITL simulations

V3- Pre-operational demonstration

Part-task HITL simulations, full-mission simulations,

V4- Factory Acceptance

Full-mission simulations, shadow-mode testing, operational trials

V5- Onsite Validation

Shadow-mode testing, operational trials

2.4

Choosing a Simulator and Assessing Fidelity

Once the validation strategy and methodology are in place, and the appropriate technique is chosen, researchers are faced with the question of which simulator(s) should be selected (i.e., offers adequate fidelity) for their validation activity. This is an important consideration since it has implications on an activity’s output, data precision, acceptability, and cost. In general, as concepts become more mature, the required fidelity of HITL exercises increases. Simulator fidelity can be divided into functional and physical aspects. Functional fidelity refers to the functions and capabilities of a simulator as compared to their counterparts of the real-world operational system that is being simulated (e.g., a fuel-burn model of a cockpit simulator). The functional fidelity is very important for fast-time simulation studies. Physical fidelity refers to the appearance and human-machine interfaces of a simulator as compared to their counterparts of the real-world operational system that is being simulated. The physical fidelity is particularly important in HITL simulation studies. HITL simulation studies involve human participants interacting with the systems or simulators. Typically, HITL simulation studies collect data on human and system performance. Therefore, it is important that the “look and feel” of a simulator is very accurate and representative of the real system. Another aspect of HITL simulation studies is the participant fidelity. If the study participants do not accurately represent the study population the results may be biased. Participant fidelity is closely related to the statistical sampling principles, e.g. study participants should closely represent the experience, age, gender, and other important attributes of the target population.

A-12

The conventional method of fidelity assessment is to classify the fidelity of a simulator as low, medium, or high. This classification is loosely based on the presence or absence of certain simulator attributes (e.g., avionics, range of motion, display capabilities, etc.). Another approach classifies cockpit simulator fidelity using defined certification levels (Level A, B, C, or D) based on characteristics and attributes. For example, a Boeing 747 cockpit simulator possessing the highest fidelity classification for its class of aircraft (Level D) is typically certified for airline training exercises. The advantage of these methods is that they are very easy to understand and apply. However, these methods neglect the fact that not all studies require the highest fidelity in all attributes, and that the required fidelity of a simulator depends on the application under investigation. For example, consider two cockpit simulators. The first simulator has six degrees of freedom for motion and the second simulator has no degrees of freedom. However, both simulators have the same avionics and the same cockpit displays. These two simulators will certainly have different fidelity for a motion sickness assessment study but will have the same fidelity for a display layout assessment study. Clearly, assessment of fidelity depends on the attributes of a simulator that are useful to the simulation objectives. It is important to realize that even if a simulator perfectly represents a real-world operational attribute (e.g., six degrees of freedom for motion), if that attribute is not required for a specific simulation application, it does not contribute to fidelity. Other methods exist to determine whether an available simulator offers adequate fidelity to conduct a particular activity. The steps below describe one method to determine the adequacy of a simulator. 1. Identify the attributes that are important to the study objectives. For example, if it is an air traffic control display simulator, it may be important to realistically represent the rate of aircraft turn, rate of climb and descent, aircraft data symbol, etc. 2. Determine the importance of these attributes in a simulation on a 1-7 rating scale. The importance rating can be received from users or subject matter experts. A rating of 1 on the scale means very low importance, 4 indicates moderate importance, and 7 indicates very high importance. The importance ratings may vary from one study to another depending on the study objectives. 3. Determine the performance of these attributes of a simulator in a test on a 1-7 scale. In order to judge the performance, a representative test must be conducted. This test will involve a study scenario. For example, an air traffic control display will involve display of aircraft operating in certain airspace. The performance rating can be received from users or subject matter experts. 4. Draw an importance-performance matrix, where importance is in columns and performance is in rows. The attributes are filled in the matrix with respect to their ratings. An example is shown below. Table 3 provides an example of an importance-performance matrix for fidelity assessment.

A-13

Table 3. Example of an Importance-Performance Matrix Performance Rating

Importance Rating 1 Very Low Importance

4

5 Moderate Importance

- Very Low Performance

- Moderate Performance

6

7 Very High Importance Climb rates

Turn rates

- Very High Performance

Aircraft data symbols

Table 3 indicates that this simulator has high performance and high importance for the presentation of aircraft data symbols. It has low performance but very high importance for the representation of aircraft climb rates. The latter observance indicates that the simulator in question is not adequate for the study. Typically, a simulator will be adequate if all of the important attributes (4 or above rating on the importance scale) have good performance (4 or above rating on the performance scale). If high importance is desired but low performance is experienced (3 or below rating), the simulator is not adequate for the application. A mathematical method based on normalization of attribute values can also be used to assess the fidelity of different simulators. The method will select a simulator based on the highest fidelity needed. This method is computationally intensive and laborious but produces a numeric assessment of fidelity. The method can be found in Kopardekar 1999.

3. BEST PRACTICES FOR HITL EXERCISES Although the sessions of the workshop from which these best practices have been assembled were organized differently, it will be more useful to the practitioner to regroup the practices according a commonly understood validation methodology. Therefore, the following sections present best practices for HITL exercises in harmony with the five steps outlined in the High Level Methodology Approach defined previously in the main body of this document (Section 5.2 of the main document). The best practices are organized by the periods before (corresponding to steps 1 and 2), during (corresponding to step 3), and after a validation activity (corresponding to steps 4 and 5). In addition, managing HITL exercises can present many challenges throughout the process, so there is a management element included in each of these sections. The intention of the authors is that these best practices serve as supplemental guidelines for experienced practitioners who perform HITL validation activities. It is also recognized that this paper may not contain an exhaustive list of all best practices associated with validation research, and that some of the best practices listed could benefit from further refinement. The original intent of the workshop was to focus only on identifying best practices for real-time HITL simulations, however, during the course of

A-14

the workshop, the practitioners agreed that many of these best practices could also apply to a variety of HITL exercises, not just real-time HITL simulation.

4. BEST PRACTICES FOR BEFORE THE HITL EXERCISE Table 4 presents a list of recommended best practices to be considered when identifying requirements, and planning and preparing for an HITL validation exercise. Each best practice (labeled as bp 1-1, bp 1-2, etc.) is described in more detail following the table. Table 4. Best Practices for Before the HITL Exercise

4.1 bp 1-1 bp 1-2 bp 1-3 bp 1-4

Managing the Process Identify the stakeholders, define their roles and responsibilities, and ensure good communication between them. Generate, compare, and prioritize lists of questions and/or concerns for each category of stakeholders. Map stakeholder questions into study requirements, then into simulation measures. Meet with controllers and pilots to discuss preliminary 'concept of use' issues before the HITL exercise, if feasible.

bp 1-5

Develop back-up plans for test design issues in the event of system problems/failures.

4.2 bp 1-6 bp 1-7 bp 1-8

Experimental Design Considerations Make a clear statement of the type of exercise that is being performed. Produce a scientific justification for the use of the chosen HITL technique. Produce a statement of any constraints that apply to the design of the experiment. Define detailed and unambiguous objectives, and state hypotheses. Keep the number of objectives to a minimum. Document those areas where the exercise needs high fidelity to the real world. Make appropriate use of baselines. Be aware of the Human Factors issues of the design. Identify all sources of error and indicate those that can and should be controlled. Airspace and Scenario Development Considerations Be open-minded. Consider unconventional options such as “generic airspace”. Research questions should drive scenario development. Identify and maintain common scenario characteristics for comparison. Starting and/or ending scenarios slowly is not usually efficient. Define and maintain necessary levels of realism. Traffic peaks and troughs may be relevant to research.

bp 1-9 bp 1-10 bp 1-11 bp 1-12 bp 1-13 bp 1-14 4.3 bp 1-15 bp 1-16 bp 1-17 bp 1-18 bp 1-19 bp 1-20

A-15

4.1

Managing the Process

The practitioner, particularly a Principle Investigator, has many responsibilities regarding managing the process of an HITL exercise. These responsibilities are essential to the success of an exercise and start early in the planning process. bp 1-1.

Identify the stakeholders, their roles and responsibilities, and ensure good communication between them

One of the major challenges involved with managing HITL activities is dealing with multiple stakeholders. A number of stakeholders can be involved in the planning, conduct, and analysis of HITL simulations, including the following: • • • • • • •

Investors/Sponsors Managers (e.g., air traffic control centers, regulatory agencies) Controllers Pilots Union representatives Suppliers Experimenters

Often a core exercise working group with representatives from each stakeholder group (or subset) is established. They actually “do the work” and facilitate the success of the exercise. The members of such a core team should be consistent throughout the duration of an exercise. It is also of paramount importance that the roles and responsibilities are clearly defined for and mutually understood by all stakeholders and core team members at the onset of the activity. When problems occur in the HITL process, they can often be attributed to miscommunication between some of the stakeholders involved. Communication at and between all levels of stakeholders is key to meeting simulation objectives and achieving a common understanding of simulation outcomes. bp 1-2.

Generate, compare, and prioritize lists of questions and/or concerns for each category of stakeholders

Each of the stakeholders has a range of concerns for an HITL exercise, some of which overlap with other stakeholders and some of which do not. Generate a list of all stakeholder questions to ensure that, for example, the appropriate simulation design is constructed, or that certain events are scripted into the test scenarios. Below is a sample of some common stakeholder questions (note: 'it' refers to whatever the simulation is testing): Management questions: •

•

Does it improve key performance areas (e.g., safety, capacity, delays, human resources issues, training costs, manpower issues, selection issues, and quality of service issues)? Is it acceptable to the participants (e.g. controllers, pilots)? Bottom Line: Will it improve matters operationally? Does it add value?

A-16

Controller/Pilot questions: • • • • • •

Will it work operationally? Will it fit in with our working methods? Will I still enjoy the job? Will it make the situation more or less safe? Can I recover if it fails? Is it a threat to my work or career? Bottom Line: Is it acceptable and useful? Does it add value?

Experimenter questions: • • • • • •

Which, of the parameters it aims to improve, have improved? Were any of the controlled parameters affected; and were any other variables affected? Is the system acceptable to controllers/pilots? Are the cognitive skills of controllers/pilots being changed in any subtle or fundamental way? Will it work in the real world? Are there any side effects? Is safety maintained, improved, or degraded? Bottom Line: What are the measurable benefits? How can we refine the design and/or operational requirements to make it better?

The HITL practitioner needs to have a common understanding of how the various questions are related to each other, where they overlap, and where they fit into the validation life cycle process. For example, a management question such as ‘Is it safe?' may cascade down to the controller/pilot level into ‘Is it safe in all circumstances, including system failures?' This question may then cascade down to the experimenter level into ‘did a loss of separation occur? Did human error occur? Was situation awareness affected? What were the effects? Did workload increase or decrease as a result?' With careful planning, all of these questions could potentially be addressed with the same set of data. bp 1-3.

Map stakeholder questions to study requirements and simulation measures

Prioritizing stakeholder questions will also assist the practitioner in planning and executing an HITL exercise. This way, if constraints occur in simulation, for example, the experimenters can make trade-off decisions to best maximize addressing the 'highest importance' management questions, controller questions, pilot questions, etc. Practitioners should ensure early on that stakeholder objectives and questions are addressed by adequate measures collected in the study. Identification of specific data requirements and the mapping of such data to objectives should be outlined in an experiment plan, before a study begins. bp 1-4.

Meet with controllers and pilots to discuss preliminary 'concept of use' issues before the HITL exercise, if feasible

Since controllers and pilots are usually the end users of the concepts evaluated in HITL exercises, their input can be an invaluable resource at the front-end of the simulation process. This stage is where techniques such as interviews with SMEs, storyboarding, and cognitive walkthroughs can be most effective.

A-17

bp 1-5.

Develop back-up plans for test design issues in the event of system problems/failures

The HITL practitioner needs to plan for the unexpected. This is particularly true with complex, larger-scale studies involving the linkage of multiple laboratories and tools, and the coordination of multiple research organizations and study participants. Having a plan in place before the HITL exercise about how to troubleshoot system problems or work around 'no-show' participants will save time when the event(s) occurs and will salvage critical data based on predetermined priorities. Such a back-up plan might involve a modified test design, alternative data collection sources, stand-by participants, and/or “buffer time” in the laboratory schedule.

4.2

Experimental Design Considerations

An individual exercise usually forms part of a research program. The experimenter needs to be clear about where their exercise fits into the wider program and should have access to all other related experimental results. When designing the exercise it must not be forgotten that it will often be run using a simulation and not on the real system, yet any conclusions drawn will be expected to apply to the real system. Ensure that the design allows this mapping of results to the real world in the areas under evaluation. The first step in the design of any experiment should be to determine the type of experiment required. Thereafter the experimental design considerations will depend primarily on the purpose of the experimental work. bp 1-6.

Make a clear statement of the type of exercise that is being performed

Types of exercises can be loosely categorized as exploratory, inferential, or demonstrative (formative or formal has also been suggested).

Exploratory work may include techniques such as pilot studies, rapid-prototyping activities or other exercises where the main objective is to show the feasibility of a potential method or approach. The appropriate use of pilot studies will prevent unexpected problems from occurring during simulation execution and analysis.

Inferential studies aim to detect differences between different systems under test. They usually have strict data requirements designed to permit statistical analysis of the results and thus require a high level of experimental control.

Demonstrative HITL simulations focus on representing the “look and feel” of the system but have few, if any data requirements. They are usually performed toward the end of the development lifecycle with the aim of confirming human involvement and commitment (e.g., user acceptability).

Though demonstrative studies are commonly used throughout the research community, for scientific reasons there should be more extensive use of inferential studies, which have more data rigor and statistical power for validation activities.

A-18

bp 1-7.

Produce a scientific justification for the use of the chosen HITL technique

A scientific justification should outline the questions that will be answered by the study and why it is felt that a particular HITL exercise is the most appropriate means to explore a hypothesis. This may be justified by first conducting cognitive walkthroughs or task analyses that identify the need for further exploration. This justification may identify further work that can be achieved using existing models or fast-time simulations and thus help reduce the scope of a real-time simulation. Where analytical models have already been conducted, these should be used to form hypotheses and predict the results of the planned simulations. A major difference between predicted results and experimental results might indicate a problem either with the model or with the simulation design. bp 1-8.

Produce a statement of any constraints applying to the design of the experiment

Having confirmed the need to undertake the HITL exercise, the next stage is to consider constraints that will be imposed on the design. A common constraint, for example, is the duration of the study, which is often limited by the availability and cost of participating controllers, pilots, laboratories, etc. Another constraint might be that the simulation pilots (also known as pseudo-pilots) were not actual pilots and therefore may have responded differently to control instructions than a certified pilot would have. It is worth considering each potential constraint and to what extent each can influence the study. Knowing limitations prior to the experiment will help shape the design and allow for planning of contingencies for capturing every essential aspect of the study. Once stakeholder requirements, the type of exercise, the reason for using real-time HITL simulation, and the list of constraints are established, the design of the experiment begins. Most of the recommended practices apply particularly to inferential studies, but should also be considered in the context of other studies types. bp 1-9.

Define detailed and unambiguous objectives, and state hypotheses.

It is imperative to produce a statement of Specific, Measurable, Attainable, ResultsOriented, Time-based (SMART) objectives. Not only should they be unambiguous but they should also map to higher level program objectives. This consideration is crucial so that exercise results actually contribute to the overall validation of the concept. Ambiguous objectives may lead an exercise astray and dilute the results. In addition, hypotheses should be stated to match expected results of the activity (which are often formulated by taking into account results of previous, related work). Occasionally, an experimenter may choose not to draw specific hypotheses. For example, this may be the case for an exercise designed to explore a new advanced concept that is not fully defined. Whether or not hypotheses are appropriate, the exercise objectives should always be clear. bp 1-10. Keep the number of objectives to a minimum It is important that stakeholders, sponsors, and experimenters understand the risks of trying to address too many objectives in a single experiment. As previously stated, it is essential to produce a statement of the exercise objectives in clear and unambiguous form. It is also important to understand that there is a strong link between the duration of

A-19

the HITL experiment and the number of objectives that can be usefully addressed. If the duration of the study is limited or fixed, then fewer objectives will result in greater confidence of results for each objective. Having too many objectives can misdirect to the focus of a exercise, put the exercise at risk for not having enough information to adequately assess a hypothesis, increase data requirements, increase the length of a study, etc. bp 1-11. Document those areas where the experiment needs high fidelity to the real world Not every aspect of a simulation needs to be of the highest fidelity. Cost and other factors often limit the level of simulation fidelity. However, it is not necessary to achieve the highest fidelity in every respect in order to relate the results to the real world. Careful consideration should be taken to ensure the level of fidelity is appropriate for each major area of the study. (See also Appendix Section 2.4 on Choosing Simulators and Fidelity Assessment). bp 1-12. Make appropriate use of baselines Baseline scenarios are often used to provide comparisons to a current operational procedure or system. However care should be taken to ensure that the simulated baseline is a sufficient representation of the real-world situation. It is also important to keep the contrasts meaningful. If the future scenario is too removed from today’s system (in time or concept of operations) then the comparison to a baseline scenario of current operations may not be valid or useful. bp 1-13. Be aware of the Human Factors issues of the design In designing the experiment, the experimenter should be aware of the impact on the human participants. An over-demanding program of runs will cause fatigue and certain invasive measures will affect performance. Ecological aspects can be used to make the experiment more natural, for example, using a simulated shift change to test for situation awareness rather than artificially stopping the traffic. Minor oversights of the design can affect results if they are not handled properly, for example provision of notepaper or pens. Such oversights will usually be detected if a pilot study is run. bp 1-14. Identify all sources of error and indicate those that can and should be controlled The main reason for using experimental design is to be able to control the sources of experimental error. Therefore the design must first identify the sources of error that are expected to be influential. Not all variance should be considered as unwanted. For example, in reality there is variance in sector entry times. If this variance is removed (as it can be in simulation) then the controller may start to ‘learn’ the traffic, which will have considerable (negative) impact on the results. It is not possible to provide an exhaustive list of sources of error and variance. The following list indicates examples of the more common ones: Traffic Familiarity: Re-use of the same traffic sample several times will introduce bias because the controllers will anticipate traffic behavior. This can be overcome by

A-20

introducing perturbations such as weather and delay but this will add variance in the traffic behavior. Sometimes it helps to change minor things that don’t affect traffic behavior such call signs or destination airports. Simulation Piloting: Pseudo-pilots are usually too compliant and their voice communications are generally clear and constant. The use of real pilots, scripts, and synthetic voice generators can improve the situation. Platform stability / fidelity: Continual interruptions due to platform problems will have a strong effect on controller/pilot attitude. If technical problems are associated with a particular period of the study this should be compensated for in the analyses. Controller/Pilot attitude: Participants in a simulation can greatly influence the results. If they are insufficiently trained on new procedures or have low confidence in a tool this will have a negative effect. Conversely, participants who have been closely involved in the development of a particular tool or concept may be overly positive/negative in their responses. Some of these individuals may be particularly useful to help plan or develop an exercise, but it is best not to use them as test subjects as they may introduce bias. Learning effects: The ability to work with new tools and procedures will almost certainly improve with time. Unfortunately, there is rarely enough time to train participants completely for advanced/future scenarios. Therefore, it is likely that participant performance in an experiment will improve with time. This can be compensated for by the use of blocking. Order of presentation: In addition to learning affects, the order in which the experimental units are presented can be influential. Randomization techniques will minimize this affect. Inter-controller/pilot variability: Differences between controllers can be compensated for by repeated measure designs. Where possible the experimenter should try to ensure that the participants are representative of the general controller/pilot population. Intra-controller/pilot variability: Variable controller/pilot performance exists in the real world. However the experimenter should be cognizant that variability may be greater for participants in real-time HITL simulation due the effects of hard-to-control factors such as traveling fatigue, unusual working hours, and extended periods away from home. Although it is recognized that intra-participant variability is difficult to control, sometimes it can be reduced by methods such as training to asymptote (i.e., training until the participants reach a bench mark criteria). It is also helpful to ensure that the simulation environment is similar to the operational environment, and that good experimental design techniques, such as randomization, are used to balance the effects of this variability.

4.3

Airspace and Scenario Development Considerations

Scenarios are usually characterized by airspace development, complexity, traffic realism, flight plan routes, traffic mix, overflights, special use airspace, weather, etc. Practitioners are challenged with creating airspace and scenarios that model the real operational environment, while addressing specific research questions.

A-21

bp 1-15. Be open-minded. Consider unconventional options such as “generic airspace”. In many HITL studies, actual airspace is replicated and real traffic scenarios are modeled to produce a familiar and realistic environment for data collection. For many exercises, using site-specific/existing airspace and traffic may be an essential requirement, such as when assessing the impact of a proposed new runway on operations at a specific airport. However, using site-specific airspace may induce constraints on the sample of subjects who participate in the study. Using generic airspace may be an option for studies that utilize many participants from various facilities or for general studies that do not need to be applied to a specific airspace. Some of the advantages of using generic airspace include the availability of a greater number of participants. Very often it is difficult to use a large number of participants from any one air traffic control facility, airline, etc. because of staffing issues or participant availability. Another advantage is the ease of participant training on generic airspace. For example, consider an exercise where a particular airspace is modeled but controllers will be brought in from various facilities. Participants who work in the facility where the airspace is located will inherently be more familiar with the airspace than those from outside facilities. Generic airspace is generally easy to learn and ensures that all controllers become trained on the airspace at the same time. This means that the level of experience is the same for all participants. Generic airspace is also more flexible than existing airspace since the experimenter has the ability to insert controlled obstacles such as weather, special use airspace, terrain, etc, and can easily create particular letters of agreement and standard operating procedures in order to address specific research questions. This approach may be very beneficial for concepts or decision support tools in the early development phase since these studies explore concepts that currently may not exist. The use of an unfamiliar airspace may help lift the participant out of present day procedures and be able to immerse them in a future concept environment. Generic airspace does have some limitations. There are difficulties with the initial creation of generic airspace. Scenarios can be more difficult to build, multi-center facilities are more difficult to develop, and time must be allocated for the development of airspace procedures. It may be difficult to baseline generic airspace scenarios and to generalize data for specific airspace. In addition, researchers will have to obtain sponsor buy-in for the use of generic airspace since results may be not be directly applied to any one facility. It is not recommended to use generic airspace when you are researching a specific procedure for a specific facility. However, if generic airspace satisfies the requirements of an exercise, be open-minded and consider the approach. Know the advantages and disadvantages, and be prepared to justify its use. bp 1-16. Research questions should drive scenario development While the practitioner should strive to develop scenarios that model the operational environment, it is also crucial to build scenarios to address the research questions and objectives. All perspectives (researcher, operational, and management) should be taken into consideration.

A-22

bp 1-17. Identify and maintain common scenario characteristics for comparison One of the biggest challenges in scenario development is to create scenarios that are experimentally comparable, but that are different enough to present a “new” problem for the participants. The experimenters often have to design scenarios that are similar (in complexity) but not exactly same. The similar scenarios are essential part of the good experimental design principles as they help statistical comparisons. However, scenarios that are too similar could produce (often undesirable) learning effects. The learning effects could negate the effects of experimental conditions. Simply changing the aircraft call signs are not always adequate since controller memory recognizes air traffic patterns as well. Learning effects may be controlled somewhat be randomizing the order of experimental conditions. There is a great need for a common and agreed upon complexity metric that practitioners can use as a way to evaluate scenarios. In this way, a particular scenario may be compared with one that may be very different in some characteristics, but has the same complexity. Using the same measure of complexity would give a standard from which to potentially compare results from various studies around the globe. Usually, similar scenarios can be created by keeping several key factors consistent such as the number of aircraft, the number of conflicts, conflict geometry, type of aircraft, callsigns, and the number and type of structured and unstructured routes. It is also best to consider the opinions of SMEs during the scenario shakedown process to ensure that scenarios are comparable but yet not the same. bp 1-18. Starting and/or ending scenarios slowly is not usually efficient A common way of building a scenario is to initiate traffic gradually into the problem, allowing the controllers to ease themselves into the scenario. When the main part of the problem is over, traffic is usually tapered off. For an hour long scenario, as much as 15 minutes on both ends of the scenario might be dedicated to “ramp up/down” time. Given the time and cost restraints to run a simulation, this is not the most effective or efficient use of scenario time. A more efficient means of building a scenario is to begin with a normal traffic load. Depending on the study, practitioners may also end the scenario in the middle of a conflict or other problem. bp 1-19. Define and maintain necessary levels of realism While developing scenarios, researchers are usually interested in maintaining high realism related to aircraft mix, traffic density, sector geometry, routes, and procedures. Many times, the scenario development process starts from collecting actual operational flight plan data. The operational data provides the realism. The operational data with realistic traffic density and aircraft types works well for near-term initiatives (e.g., Reduced Vertical Separation Minima). However, for future concepts such as free maneuvering, operational data may need considerable modifications in order to satisfy the experimental objectives. In such situations, a number of variables (such as future traffic load, future aircraft mix, and technologies) may need to be modified. Such modifications to better model the future environment may not be realistic in the present day operations, however will represent the future environment. Care must also be taken during the conduct of the simulation. For example, if a scenario is developed such that it begins with a normal or heavy traffic load, practitioners may

A-23

consider utilizing subject matter experts to brief a participant controller onto position. Since this is a normal operation for controllers, this would be an acceptable way to start the run with a full traffic load. bp 1-20. Traffic peaks and troughs may be relevant to research HITL practitioners often try to impose high workload levels on simulation participants to stretch the limits of the participant’s abilities, and to test the limits of new concepts and procedures on the air traffic system. To do this, they typically design traffic scenarios to represent peaks or high levels of traffic activity. Slower-manifesting problems are generally avoided since the practitioner wants to make the best use of valuable time. Prior simulation research has shown, however, that operational errors often occur in the beginning of troughs, or lower levels of traffic, immediately following very high levels of traffic activity (this results because of a temporarily perceived reduction of complexity and thus lower vigilance). In order to better emulate the operational environment and to capture all possible conditions for human error, HITL practitioners should script a range of traffic activity into their scenarios; those which include both peak levels of traffic and troughs.

5. BEST PRACTICES FOR DURING THE HITL EXERCISE Table 5 presents a list of recommended best practices to be considered when carrying out the tasks of an HITL validation exercise. Each best practice is described in more detail following the table. Table 5. Best Practices for During the HITL Exercise 5.1 bp 1-21 bp 1-22 5.2 bp 1-23 1-24 1-25 3 1-26 1-27 1-28 bp 1-29 bp 1-30 5.4 bp 1-31

anaging the Process not sacrifice simulation quality for the interest of time. cument unforeseen effects of test variables on system performance not captured by system measures. perimental Design Considerations sure adequate training of participants. prepared to administer contingency plans if necessary. vestigate concomitant measures carefully. bjective and Objective Data clear as to what is meant by ‘subjective data’. e objective data to clarify subjective findings (and vice versa). plain to the participants why their feedback is needed and how it will be used. derstand the factors that could influence subjective data. nimize the impact of external factors on subjective data. lidation Data Storage e a central data repository such as the Validation Data Repository.

A-24

5.1


bp 1-21. Do not sacrifice simulation quality for the interest of time A small set of good data is better than a large set of bad data. Whatever the time constraints may be, the HITL practitioner must remember this guideline and plan accordingly. bp 1-22. Document unforeseen effects of test variables on system performance not captured by system measures Though they may not be related directly to study objectives, researchers must be cognizant of unforeseen or subtle effects of simulation variables. Practitioners should document and report all observations and results, expected or otherwise, so that that may further educate and assist future management decisions.

5.2

Experimental Design Considerations

bp 1-23. Insure adequate training of participants The value of training participants in an HTIL activity is often underestimated. It is generally desirable that participants clearly understand the objectives and the design of the experiment before they actually participate. This is often achieved by briefing participants prior to the start of an activity. Most importantly, they must have sufficient laboratory familiarization and be adequately trained on any new procedures, equipment, etc. to ensure the validity of the results. When schedules are condensed, often the first thing to be reduced is the amount of time allotted for participant training. Do not compromise this aspect of simulation. bp 1-24. Be prepared to administer contingency plans if necessary. As mentioned in bp 1-5, it is a fact of life, particularly with real-time HITL exercises that some data or recordings may be lost due to technical reasons. The exercise should have been designed to accommodate a certain number of these losses without too much detriment to the exercise. The entire research team and laboratory personnel should be aware of the contingency plans in advance and be prepared to execute them should they be necessary. bp 1-25. Investigate concomitant measures carefully. The choice of measures (dependent variables) and experimental factors (independent variables) will depend on the objectives, and in turn the objectives should depend on the existence of suitable variables. The number of concomitant measures should be kept to a minimum to avoid the risk of finding significant effects that contradict simply by chance. It is difficult to explain such observances, especially to the stakeholders. However, it is a delicate balance because when several concomitant measures all indicate the same effect, this adds to the confidence in the results.

A-25

5.3

Subjective and Objective Data Collection

bp 1-26. Be clear what is meant by ‘subjective data’ In any experimental work it is important to be clear about the type of data that is being collected. The experimenter should consider first whether any experimental benefit is to be gained from distinguishing between subjective and objective data types. In many cases these terms are misused to express the difference between qualitative and quantitative data. In the particular case of HITL studies, it can be useful to distinguish between the objectively measured system variables and those variables derived from the subjective opinions of the participants. An alternative terminology could be to define perception measures (subjective) and observed measures (objective). This definition also helps to clarify that the subjectivity is that of the human participant and not that of the observer. bp 1-27. Use objective data to clarify subjective findings (and vice versa) Clearly differences between the perception of the participants and the observed system recordings are of great interest to the experimenter. For this reason both types of data should be collected. In evaluation studies objective data can be used to generate discussion with the participants after an experimental run. Similarly subjective data can be aggregated statistically to produce classifications, for example, defining a “high workload” traffic sample based on subjective ratings. bp 1-28. Explain to the participants why their feedback is needed and how it will be used. The extent to which the subjective results will be used should be explained clearly to the participants. If they feel they are being asked to “sign off” on a new system, their feedback will be more guarded (and less constructive) than if it is made clear that the development is still in the early stages. bp 1-29. Understand the factors that could influence subjective data The attraction of subjective data is clear. The simplest way to find out if a new system is suitable is to ask the participants directly. Indeed in certain aspects (e.g., trust and confidence), subjective measures are the only possibility. However subjective data needs to be treated with great caution as it is very easily corrupted by external influences. The experimenter must not simply obtain the opinion; they must also understand where it comes from. It is essential to ensure that the participants have received sufficient information and training on the test system in order to give an opinion. Most participants will attempt to answer the questions put to them and may not be able to judge whether they have sufficient understanding to do so. They may also be reluctant to be too critical about the system they are asked to evaluate. Experimenter bias can have a big effect here. The more the experimenter is seen to promote the system, the less the participants will be likely to criticize it. First impressions are particularly important: an opinion formed on improper use of a tool may be very difficult to change later on. Moreover, individuals will

A-26

inevitably be influenced by the comments of their colleagues. In group situations there can be a tendency not to want to oppose the majority view. bp 1-30. Minimize the impact of external factors on subjective data Subjectively expressed opinions are also affected by external factors such as fatigue, platform performance, and mood. Therefore the experimenter should develop sets of probing questions to ensure that the opinions expressed are based on a sound understanding of the system and that there is minimal influence of external factors. Subjective variables are in general less suitable for repeated measures. If the same questionnaire is presented after each experimental run there will almost certainly be some degradation in the quality of response given. Care must be taken not only in the wording of questions but also in the timing and frequency with which they are asked. A final point to note in collecting and relying on subjective information. It is clear that air traffic controllers like to solve complex problems (e.g., sequencing, conflict detection and resolution) and do not like the routine tasks such as data block management and data entry. Therefore, any concept that will eliminate or reduce job satisfaction may face resistance. Hence, researchers should be cautious in designing, conducting, and analyzing the results involving such concepts.

5.4

Validation Data Storage

Good data recording and storage are an important feature of any experimental work. In addition to the experimental results obtained, it is essential to keep records of all input data i.e., experimental conditionals, participants, methods, limitations, and any circumstantial information which could influence the interpretation of results. This can be done by maintaining a simple log book, however, the use of electronic formats will make retrieval and dissemination easier. bp 1-31. Use a central data repository such as the Validation Data Repository The collection, retention, and availability of validation data is becoming exceedingly important. The primary benefit of a central data repository, such as the Validation Data Repository (VDR), is to collect this data in one place, where it can be easily searched, retrieved, and analyzed. The use of a central data repository will support: • • •

Internal project communication – all actors have access to the same data Stakeholder communication – for review and dissemination Publication – peer group review and publication of results, methods, techniques, tools, scenarios, etc.

The VDR, currently under development, is described in the main body of the document in Section 5.4.

A-27

6. BEST PRACTICES FOR AFTER THE HITL EXERCISE Table 6 presents a list of recommended best practices to be considered when analyzing results and preparing the report for an HITL validation exercise. Each best practice is described in more detail following the table. Table 6. Best Practices for After the HITL Exercise 6.1 1-32 1-33 1-34 1-35 1-36 6.2 1-37 1-38 1-39

6.1

anaging the HITL Process port all results, not just those that show an effect early present results to management personnel. ovide results in a timely manner ef results to participating controllers and pilots. llow process through to implementation. atistical and Operational Significance sure that the analyses and results are operationally relevant nly use inferential statistics for pre-planned comparisons atistical significance doesn’t always imply operational significance


Regardless of the expected or desired results of a validation activity, it is the obligation of the practitioner to responsibly analyze all data and present all results. bp 1-32. Report all results, not just those that show an effect Even if many dependent variables are used, report all results, not just those that show an effect. Often results of data analyses that show no effect are just as important when operationally interpreted. Regardless, it is good practice to be thorough when reporting the results of data. bp 1-33. Clearly present results to management personnel Results should be presented to management personnel in clear, non-technical terms. In order to convey relevant information, it is important to understand and keep in mind the goals of management. Requesting that management reiterate their interpretation and understanding of the results in their own words will help ensure that there is no misconception of the results and conclusions of the exercise. bp 1-34. Provide results in a timely manner Practitioners recognize the need to provide expeditious analyses to stakeholders and sponsors, but often it is not possible to develop the final report with extensive analyses quickly. Under such circumstances, consider producing two reports, a “quick-look report”, and the standard final report. For example, the quick-look report for a fullmission real-time simulation could be primarily based on observations made during the exercise, and the results of questionnaires, interviews and debriefings with the participants. The final report would be expanded to include all sources of data, detailed analyses, detailed results, recommendations, etc. Caution must be exercised in

A-28

producing a quick-look report. The practitioners and SMEs must carefully consider any interpretations presented since they are based on limited data. bp 1-35. Brief results to participating controllers and pilots After an HITL exercise concludes, practitioners typically do not have any further communications with their study participants. Because data analysis and report generation is often a time-consuming process, this dissociation grows even further by the time practitioners are prepared to release the results. The fact is, however, that controllers and pilots are usually very interested in knowing the outcomes of the studies in which they participated. They benefit from knowing results because they are essentially spokespeople and information resources on the topic of the evaluation at their respective facilities. In some cases, they may even be able to assist in the training of a new concept or tool because they have had first-hand experience using it in a simulated environment. HITL practitioners should survey their study participants for interest in obtaining documentation or a briefing at the end of the research and follow through with those requests. Not only will it benefit the controllers and pilots, it will also strengthen the link between the research and operational communities. bp 1-36. Follow process through to implementation If possible, HITL practitioners should track the progress of the concept explored in their exercise to see if and how it gets implemented. They should identify themselves and/or their organization as a resource for information. Decision-makers further down the line are often not aware of or not knowledgeable on the prior testing and may have questions pertaining to the exercise.

6.2

Statistical and Operational Significance of Results

Researchers are expected to indicate the relevance or impact of the results of an HITL exercise in operational terms. In the early stages of an operational concept, data from an HITL exercise is often more descriptive in nature and does not lend itself to abundant statistical analyses. At this stage, results often focus on providing insight into the development of the concept itself and potential operational considerations such as procedural needs, information needs, and impact on human performance. As concepts mature, statistical analyses of data from HITL exercises become more important since quantifiable benefits in terms of safety, capacity, delays, etc., must be also provided by validation exercises. Such analyses require greater scientific rigor. bp 1-37. Ensure that the analyses and results are operationally relevant It is very rare to conduct one simulation exercise to solve complex operational issues or problems. Often a series of exercises are necessary to address some operational decisions. It is critical that the analyses and results of multiple exercises be traced to clear high level validation objectives that relate to operational considerations such as feasibility, safety, benefits assessments, etc. Clear traceability of results from an exercise to relevant operational objectives is necessary.

A-29

bp 1-38. Only use inferential statistics for pre-planned comparisons In inferential studies, statistical significance helps the experimenter to make comparisons regarding the validity of their experimental hypothesis. Good statistical practice and risk of error dictate that only comparisons planned before the exercise should be considered. If exploratory analyses are used, and interesting comparisons are identified, post developed hypotheses and inferential statistics should not be applied due to the risk of bias and misinterpretation of results. Such valuable information should be used to plan further experimentation. bp 1-39 Statistical significance doesn’t always imply operational significance The fact that statistical significance is found does not necessary mean that results translate to meaningful operational differences. For example, consider a case where the workload between two procedures is found to be statistically significant but in both conditions it is very low and would not lead to any operational concerns. The statistical significance will tell you if there is a difference but it will not indicate how meaningful the difference is from the operational perspective. Therefore, statistical significance must be treated with caution. Often, perception (subjective) responses gathered from questionnaires, interviews, and debriefings help to sort out the operational relevance of observed (objective) results. Therefore, HITL exercises should include subjective data collection strategies designed to substantiate and to interpret the results of the objective tests.

7. SUMMARY OF THE HITL EXERCISE DEVELOPMENT PROCESS The development of an HITL exercise is a careful process with many activities that should follow a defined methodology. These activities are described in the context of the High Level Methodology Approach described in the main document: Step 1. Identify the requirements • Define study specific objectives: A clear and concise statement about the objectives will be developed. • Form a team: The team will consist of members from the operational concept validation management and system analysis team, SMEs, union representatives, researchers, statisticians, human factors engineers, sponsors, and other members. It must be emphasized that all stakeholders must be represented in this team. • Identify the type of study: The team will identify the suitable type of study (e.g., paper study, fast-time simulation, real-time simulation, or rapid prototyping) necessary to accurately assess the operational and technical feasibility of the proposed system changes. Step 2. Prepare the Validation Plan • Develop experiment plan: An experiment plan detailing the background, objectives, literature review, procedure, data collection and analysis methods, and schedule will be developed. • Develop detailed metrics: The team will identify, define, and develop, as necessary, the metrics required to support the objectives of the study.

A-30

•

Develop scenarios and select equipment: The team will develop air traffic scenarios and select the equipment (e.g., simulator) with due consideration to fidelity requirements. • Schedule laboratory and support personnel: Team will conduct the necessary coordination to ensure that adequate laboratory time is available and support staff will be available when required. This step is typically only required for real-time, HITL simulation studies. • Conduct readiness/shakedown testing: Trial runs of the scenarios will be conducted to ensure that the scenarios, laboratory environment, and operations are realistic. This step is typically only required for real-time, HITL simulation studies. If necessary, various laboratories need to be integrated and configured to suit study objectives. Step 3. Carry Out the Validation Plan • Conduct simulation and collect data: Members of the team will conduct the study and collect the data as outlined in the experiment plan. Step 4. Analyze the Validation Results • Analyze data and develop recommendations: Once the data is collected, members of the team will analyze the data and develop recommendations. Results from multiple studies aimed at evaluating a single operational concept will be reviewed and merged to form a list of recommendations. Step 5. Prepare the Validation Report • Prepare and Publish Reports: All data, results, and recommendations will be provided to sponsors and other interested parties via published reports. • Provide data/information to a central data repository: Information concerning the exercise will be incorporated into the VDR. Different researchers and organizations will share such VDR information to facilitate validation activities.

8. OVERALL SUMMARY The first U.S.-Europe workshop sponsored by FAA/EUROCONTROL Action Plan 5 produced several recommended practices related to HITL exercises. The following topics were discussed in the workshop: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Role of HITL exercises in the validation process, General overview of HITL simulation process, Managing HITL exercise, Scope and fidelity considerations, Experimental design considerations, Airspace and scenario characteristics, Subjective and objective data collection, Sources of error and variance, and Statistical and operational significance of results.

Discussion of the first topic captured how HITL exercises could be used for the validation process. The topic on the general overview of the HITL simulation process discussed when, how, and why HITL exercises need to be conducted. The topic on managing HITL exercises provided the practitioners’ perspective about how to ensure that HITL exercises address higher-level management questions, researcher questions, and operational questions within the constraints of available resources. The topic on scope

A-31

and fidelity considerations provided information on how to decide the scope of an exercise and how to ensure and assess adequate fidelity. The topic on experimental design considerations highlighted experimental protocols and considerations particularly applicable to HITL exercises. The topic about airspace and scenario characteristics reviewed the use of generic and real airspace, and highlighted good practices for creating traffic realism and managing scenario characteristics. The topic concerning subjective and objective data collection reviewed the options of when and how to use the different types data. The topic concerning sources of error and variance identified those areas which should and should not be controlled for in simulation practice. The final topic on statistical and operational significance covered how to report results and interpret operational meaning from statistical information. From the topics discussed by the practitioners in the workshop, there were many good practices gathered which are reported according whether they apply before, during, or after an HITL exercise.

A-32

References Federal Aviation Administration (1999). Human Factors Job Aid. Federal Aviation Administration (1999). Operational Concept Validation Process. FAANational Airspace (NAS) Advanced Concepts Branch –ACT 540. Kopardekar, (1999). Simulation Fidelity, Published in Industrial and Occupational Ergonomics: Encyclopedia, ISBN 0-96545-6-0-0.

A-33


Appendix B Best Practices for Human-in-the-Loop Exercises

Prepared By: Andrew Harvey, Conor Mullan EUROCONTROL Research Centre Albert Schwartz, Sherri Magyarits Federal Aviation Administration William J. Hughes Technical Center

20 November, 2003

1

Acknowledgments The persons listed below attended the March 2003 workshop in Rome, Italy. This paper is a collection of their ideas, experience, and expertise. The authors would like to thank the attendees for their dedicated involvement and contributions to this effort. AENA, Spain: José Miguel de Pablo, Manuel Dorado BAE Systems: Washington: Donald Eddy CENA, France: Caroline Chabrol, Yann LeFablec, Vincent Kapp, Didier Pavet CNA, Washington, D.C.: Joseph Post CSSI, Washington, D.C.: Stephane Mondoloni Deep Blue, Italy: Marinella Leone, Patrizia Marti, Alberto Pasquini DFS, Germany: Andreas Tautz DGAC, France: Bernard Bonnafoux DLR, Germany: Franz Knabe Eurocontrol Experimental Centre, France: Peter Crick, Andrew Harvey, Yann Kermarquer, Nigel Makins, Conor Mullan Eurocontrol HQ, Belgium: Ulrich Borkenhagen, Keith Vickery, Hans Wagemans ENAV, Italy: Giancarlo Ferrara. Christiana Abbate, Giorgio Matrella, Francesco Podiari ERAU, Florida: Ian Wilson FAA Headquarters, Washington D.C.: Dino Piccione, Diana Liang, Sharon Kurywchak FAA William J. Hughes Technical Center, NJ: Karen Buondonno, Parimal Kopardekar, Albert Schwartz Fraport, Germany: Oliver Dellman ISA Software, France: Ian Crook LMI, Virginia: Shahab Hasan MITRE, Virginia: Fred Wieland, Urmila Hiremath NASA: Sandy Lozito, Karlin Roth, Gary Lohr NATS UK : Joanne Thornton NLR, Netherlands: Juergen Teutsch OECD- HRP, Norway: Gyrd Skraaning SICTA, Italy: Roberto Caramanica, Giovanni Esposito, Rossana Loffredo SJSU, California: Kevin Corker TransSolutions, Texas: Belinda Hargrove University of Sienna, Italy: Antonio Rizzo Volpe, Massachusetts: Kim Cardosi

B-2

Table of Contents 1 Background ...............................................................................................................6 2 Second US/Europe Practitioners’ Workshop on best practices in the development of fast-time and real-time simulation scenarios for validation activities. ..............................6 3 Relevant Terms & Definitions ....................................................................................7 3.1

Simulation Definitions ...................................................................................7

3.2

Scenario Definitions......................................................................................7

3.3

Scenarios in Simulations...............................................................................8

4 Scenario Uses ...........................................................................................................9 4.1

Scenarios and Levels of Maturity ..................................................................9

4.2

Scenario Objectives....................................................................................10

5 Steps in Scenario Design ........................................................................................11 6 Coordination of Fast & Real-Time Simulation Activities ...........................................11 6.1

Data Sharing and Re-Use between Fast-Time & Real-Time .......................12

6.2

Technology Advancement ..........................................................................14

7 Scenario Development Considerations....................................................................14 8 Traffic Building.........................................................................................................18 9 Specific Event Scripting...........................................................................................20 10 Validation Data Repository ......................................................................................21 10.1 How the VDR Can Help ..............................................................................21 10.2 How the VDR Treats Scenarios ..................................................................22 11 Summary of Best Practices in Scenario Design.......................................................23

B-3

List of Tables and Figures 1 - Best Practices for Coordination of Fast & Real-time Scenario Development ..................12 2 - Best Practices for Scenario Development and Design ...................................................15 3 - Best Practices for Building Traffic...................................................................................18 4 - Best Practices for Scripting Specific Events ...................................................................20 Figure 1 - Iterative nature of fast- and real-time analyses. ...................................................12 Figure 2 - The role of scenarios in the VDR.........................................................................23

B-4

Executive Summary The second Federal Aviation Administration (FAA)/EUROCONTROL Action Plan 5 Validation and Verification Strategy Workshop, held in conjunction with Action Plan 9 Air Traffic Modeling of Operational Concept, was conducted on March 11-13, 2003, to develop and standardize ‘best practices’ related to the development of simulation scenarios for validation exercises. The workshop was well attended by 32 European and 19 United States (US) practitioners, all of whom were experienced in fast-time and real-time simulation for concept validation. The workshop focused on the development of simulation scenarios for both fast-time and real-time simulation and covered the following topic areas: 1. Coordination of Fast & Real-time Simulation Activities 2. Scenario Uses 3. Scenario Considerations 4. Traffic Building 5. Specific Event Scripting 6. Analysis Considerations Practitioners introduced each of the topic areas with a short briefing, which was then followed by a discussion session facilitated by a moderator. After the initial discussions on each topic, practitioners were divided into working groups to further discuss and identify elements of best practices for the topics to which they were assigned. Their consolidated recommendations were presented to all workshop participants for final discussion and consensus. The resultant best practices for developing and validating simulation scenarios are presented in this appendix. Their purpose is to serve as supplemental guidelines for experienced practitioners who perform concept validation activities. Additional material concerning scenario development and coordination of fast and real-time activities outside the scope of the topic areas listed above is also included in this appendix, based on discussions held during the workshop. Overall, the participating practitioners felt that the workshop was very valuable and useful for their research endeavors. They suggested that further workshops be arranged to discuss additional topics such as Metrics and Measures, more specific topics on Scenario Development, and Reporting Results.

B-5

1. BACKGROUND The Federal Aviation Administration (FAA)/EUROCONTROL R&D Committee was established in December 1995 during the second FAA/EUROCONTROL R&D Symposium, held in Denver, Colorado. The focus of the FAA/EUROCONTROL R&D Committee was to define priorities in terms of common actions and agendas of both organizations. The Committee identified areas of mutual interest where the FAA and EUROCONTROL could work together in R&D and defined several R&D Cooperative Tasks, which are referred to as ‘Action Plans’. The goal of Action Plan 5 (AP5) is to determine a unified strategy for validating and verifying the performance, reliability, and safety of Air Traffic Management (ATM) systems. One objective of Action Plan 9 (AP9) is to promote a mutual understanding between the United States (US) and Europe on the use and development of fast-time simulation models for modelling of air traffic operational concepts. An objective shared by AP5 and AP9 is to develop detailed best practices for performing tasks associated with the verification and validation of ATM systems. The development and subsequent use of these best practices will allow for sharing of information and comparison of results among all researchers. Best practices covering the use of metrics, data collection, data analysis, and reporting are useful for each type of validation exercise, as they take into account several factors including resources, cost, and most importantly, prior 'lessons learned'. In order to capture ‘lessons learned’ from a range of research organizations, several workshops were conducted and more will follow. The attendees of these workshops include experienced practitioners in their respective fields and in the topics presented at the workshops. This appendix describes the best practices resulting from the Second US/Europe Practitioners’ Workshop, where the topic for discussion was the development of fast- and real-time simulation scenarios for validation activities.

2. SECOND US/EUROPE PRACTITIONERS’ WORKSHOP ON BEST PRACTICES IN THE DEVELOPMENT OF FAST-TIME AND REAL-TIME SIMULATION SCENARIOS FOR VALIDATION ACTIVITIES. The FAA and EUROCONTROL organized the second AP5 workshop in conjunction with AP9. The workshop took place in March 2003 in Rome, Italy. The objective of this second workshop was to discuss and identify best practices from practitioner experience in the development of simulation scenarios for validation activities. During ATM concept validation, both fast-time and real-time simulation techniques are used. These techniques require the development of scenarios, but at present, no standardized guidelines exist to steer the researcher through the development of these scenarios. During this workshop, researchers identified best practices based on actual experiences in developing scenarios. The best practices presented in this document pertain to both fastand real-time techniques. Those that are unique to fast-and real-time techniques are identified as such. In addition, relevant best practices from the Operational Concept Validation Strategy Document (OCVSD) Appendix 1 ("Best Practices for Human-in-the-Loop Validation Exercises"), hereafter referred as Appendix 1, are also included where appropriate. The Second Workshop participants included 19 practitioners from the US and 32 practitioners from Europe representing both fast-time and real-time simulation fields. The

B-6

US participants included researchers from the FAA William J. Hughes Technical Center, National Aeronautics and Space Administration (NASA) Ames Research Center, NASA Langley Research Center, Volpe Transportation Research Center, MITRE, BAE systems, Embry-Riddle Aeronautical University (ERAU), Logistics Management Institute (LMI), CNA Corporation, CSSI, San Jose State University, and TransSolutions. The Europe participants included researchers from Aeropuertos Españoles y Navegación Aérea (AENA), Deutsche Flugsicherung GmbH (DFS), Direction Générale de l'Aviation Civile (D.G.A.C.), Sistemi Innovativi per il Controllo del Traffico Aereo (SICTA), Deutschen Zentrum für Luft- und Raumfahrt (DLR), EUROCONTROL, EUROCONTROL Experimental Centre, Frankfurt Airport Services Worldwide (Fraport), Deep Blue, Ente Nazionale di Assistenza al Volo (ENAV), Centre d’Etudes de la Navigation Aérienne (CENA), ISA Software, National Air Traffic Services (NATS), OECD Halden Reactor Project, and Nationaal Lucht en Ruimtevaartlaboratorium (NLR).

3. RELEVANT TERMS & DEFINITIONS 3.1

Simulation Definitions

Because fast-time and real-time simulations are distinct approaches, each technique has its own terminology, data, and validation strategies. To clarify some of the differences, this section contains a definition of terms. 3.1.1

Real-Time Simulation

A real-time simulation is one in which a human operator (e.g., air traffic controller) interacts with and reacts to simulated conditions in near real-time. 3.1.2

Fast-Time Simulation

A fast-time simulation is one in which there is no interactive human involvement in simulated conditions. Instead, scenarios unfold using rule-based decisions that control the interactions between simulated actors. Note: There are hybrid simulation techniques, which allow human interaction, but do not have to run in real-time. For purposes of this appendix, these techniques are classified under fast-time simulation.

3.2

Scenario Definitions

Scenarios are well recognized as an important tool for assessing the impact of proposed changes on the ATM system, and in the case of human-in-the-loop studies, on human performance. There are different views on the definition of scenario, depending on the context in which the term is being used. The following paragraphs define the term ‘scenario’ within various contexts. In ATM, the two most common definitions refer to Operational Concept Scenarios and Validation Scenarios. 3.2.1

Operational Concept Scenario

An operational concept scenario is a documented description of a sequence of events involving one or more ‘actors’ that is focused on some specific ATM function or procedure. This type of scenario is generally implemented during initial concept design and

B-7

development phases. It allows various instances to be described during the identification and refinement of issues for further testing and development. An operational concept scenario used in concept design can be used to describe various “what if” scenarios in order to judge or explain how a concept should work in these instances. Such a description is likely to be generic in nature, not requiring specific details of the environment. The following example is taken from a Principles of Operations document for an En-Route concept using System Supported Co-Ordination (SYSCO). While the example is detailed, the generic description and general applicability of this description should be noted: “Sector 1 is an enclosed sector bordered on either side and to the south by sector 2, and on either side and to the north by sector 3. All sectors are en-route sectors; S1 has a TMA below for traffic routing to/from a small airport at point C3. Aircraft ABC1 has a planned routing from point A1 along route A to exit the sector at point A4, continuing on to point A5 and beyond. Aircraft ABC2 has a planned routing from point B1 to point B3, before joining route C to land at point C3. Both aircraft are cruising at their RFL of FL340 and are of similar type. The scenario begins with ABC1 in S2 approaching S1, and with ABC2 in S3 approaching S1. Prior to sector entry S1's system has received, via SYSCO messages, advanced warning information of ABC1 & ABC2 from S2 & S3 respectively. Both upstream sectors have a LoA with S1 and both aircraft are complying with the conditions of the agreement. Because of this, the SYSCO systems have automatically generated and agreed upon entry/exit conditions compliant with the LoA. Therefore, no explicit co-ordination task has taken place and the system dialogue has remained transparent to both sets of sector controllers” 3.2.2

Validation Scenario

A validation scenario is an extension of the operational concept scenario. It is a representation of the events, actors, and interactions of the operational scenario applied in a simulation environment. The objective is to excite the performance and interactions described or expected in the operational scenarios. The simulation environment refers to various configurations of airspace, traffic sample, weather, failure modes, and any other controllable variables that might affect the performance of the ATM system. In this way, a validation scenario will test the assumptions in the concept scenarios and thus the concept design. The validation scenario is that which is actually implemented in a simulation in order to provide robust feedback in terms of whether or not a proposed concept can be implemented in the operational environment. Validation scenarios are actually run in the experiments, therefore they will require specific traffic files, airspace files, and possibly scripted events in order to execute properly. They will also have associated validation objectives and be able to address specific indicators and metrics.

3.3

Scenarios in Simulations

There are several types of scenarios; each takes on a different meaning. The following definitions distinguish the various scenario terms.

B-8

3.3.1

Real-Time Simulation Scenario

A scenario for a real-time simulation is generally characterized by a specific or generic geographical environment, traffic sample, and special scripted events, such as weather. A variety of scenarios may be developed for a real-time activity in order to gain generality in the results. For example, traffic samples may vary by volume to test the limits of the human operators with new concepts, procedures, and/or equipment. In general, scenarios in realtime simulation are meant to challenge the human operator in some or all aspects of the system. They are used to allow for measurement of workload, for example, of a new concept on an air traffic controller. 3.3.2

Fast-Time Simulation Scenario

A scenario for a fast-time simulation is characterized by a given set of conditions that include a specific geographic environment (e.g., airspace, airport), traffic sample, event or set of events, procedures, and any other controllable variables (e.g., weather, failure modes) that are of interest to the outcome of the simulation. Scenarios in fast-time simulation typically represent normal operating conditions of a new concept for analysis. In other words, if a new procedure is being evaluated, the fast-time model runs assuming the procedure performs without fail throughout the scenario. This is an effective approach when predicting potential capacity gains, for example, of a new concept. 3.3.3

Baseline Scenario

A baseline scenario is a reference scenario that provides a benchmark against which effects of the experimental conditions (i.e., proposed operational concepts) can evaluated. The baseline scenario establishes a point of comparison. It implements scope and control conditions for testing of the hypothesis. It does not implement experimental conditions.

the be the the

4. SCENARIO USES Scenarios are a core component of many types of evaluations and simulations. Their application is not limited to any one type or level of testing. Rather, they can provide the basis for exploratory studies, demonstrations, usability testing, and operational test and evaluation activities. The objectives of scenario use and the actual way in which they should be used can change with increasing maturity in the concept development. This section begins by discussing the use of scenarios in the typical Concept Validation stages (V1 -V3) and is based on the levels of maturity outlined in the OCVSD.

4.1 4.1.1

Scenarios and Levels of Maturity V1 Establishment of Concept Principles

In the earliest stages of concept development, the concept designers will not have a wellformed idea of the concept principles or operational and technical requirements. The scenarios at this stage will be focused on understanding the potential of the concept. The purpose at this stage is to help form the principles, eliminate poor design choices, and focus on refining the concept. Scenarios will be focused on establishing the scope of the concept.

B-9

4.1.2

V2 Proof of Concept

In the next stage of development, basic assumptions should already have been tested and basic design already formulated. At this level, scenarios should focus on more advanced aspects of the concept design, proof of use, non-nominal cases or marginal capabilities. At this stage, there may still need to be a repetition of the type of analysis performed after V1, but mainly the scenarios here will not be so singularly focused. The outcome of this stage in the validation process should be a mature, stable concept design with an initial proof of concept. Scenarios will be focused on setting the limits of the concept, establishing procedures and phraseology and determining clear requirements to assist in producing a stable environment for the final pre-implementation phase. 4.1.3

V3 Concept Integration and Pre-Ops Trials

In the final stage of the concept development, the scenarios will focus on repetition, replication, and performance measurement. The experiments here will be assessing a mature concept in rigorous, high fidelity experiments (large fast-time or real-time). As such, the scenario will need to be designed with choice of indicators and metrics in mind. Large detailed scenarios will be built to model the whole system under assessment. The scenarios will need to be run repeatedly in order to generate enough data for performance measurement, with baseline scenarios taken into consideration. The outcome of this stage in the validation process will be detailed information about the expected performance of the system under a variety of normal conditions and the ability of the system to cope with various non-normal conditions.

4.2 4.2.1

Scenario Objectives Exploration

Scenarios can be used to develop and refine a concept and explore what-if questions. Methods suitable to meeting these objectives include cognitive walkthroughs, part task simulations, and fast-time simulations. Exploratory scenarios are generally a low-cost means of identifying issues and refining the experimental approach for future studies on a proposed operational concept. 4.2.2

Demonstration

Scenarios can be used to show or exhibit a potential capability to a targeted audience. They can be incorporated into demonstrations, which although do not prove a concept, can give interested parties a view of how a concept might appear in the operational environment. With demonstrations, researchers may have to forego strict experimental design – e.g., procedures may evolve as demonstration progresses, but this does not undermine the value of the information gained from the demonstration. 4.2.3

Usability Testing

Scenarios created for usability testing can help to determine the appropriateness of a proposed tool, procedure, and/or equipment. Usability testing typically examines user preference and performance for a specific use, and thus scenarios are generated correspondingly to address issues associated with the proposed concept.

B-10

4.2.4 Operational Trial Operational trial scenarios are used to confirm that the ‘system’ under test performs to predetermined standards and/or criteria when stressed. The operations trial scenario is designed to elicit performance to the level that it demonstrates the system works as expected under all foreseen exceptions and failure modes. It is also designed to answer questions related to the expected benefits and expected costs of implementing a new concept. A trial may involve a variety of scenarios in order to gain generality and statistical relevance in the results. The number of scenarios or number of concept relevant events in those scenarios needs to be derived from the required statistical relevance.

5. STEPS IN SCENARIO DESIGN The following outline reviews the basic steps to be carried out when designing scenarios. The steps are iterative in nature, and as such should be continually applied as the concept matures. 1. Clearly define simulation objectives − Consider concept level of maturity − Consider operational understanding − Obtain stakeholder input 2. Identify and prioritize appropriate metrics/ evaluation criteria 3. Determine relevant scenario characteristics − Bear in mind that scenarios should cover a variety of dimensions: normal to abnormal, simple to complex, low to high taskload, and procedural tasks versus problem solving −

Consider including a wide range of scenarios to allow for greater generalization of results

−

Consider a range of scenario characteristics, including the following: o Domain o Scope o Duration o Fidelity o Focal points o Experimental validity o Human performance requirements Obtain stakeholder input

−

4. Determine assumptions and limitations 5. Identify data management and analysis requirements 6. Prepare detailed reports − Document assumptions and limitations and their implications on the results − Obtain stakeholder input

B-11

6. COORDINATION OF FAST & REAL-TIME SIMULATION ACTIVITIES Table 1 presents a list of recommended best practices to be considered when coordinating scenarios between fast-time and real-time simulation activities. Each best practice is described in more detail following the table. Table 1 - Best Practices for Coordination of Fast & Real-Time Scenario Development ordination of Fast & Real-Time Simulation Activities mmunicate with fellow fast-time and real-time researchers arly state and document your design assumptions arly state the likely implications of your assumptions Standardize scenarios for use in both fast- and real-time simulations when possible

6 bp 2-1 bp 2-2 bp 2-3 bp 2-4

Traditionally validation has taken the approach suggested in the Validation Route Map, that first you do fast-time, followed by a real-time simulation (see Section 5.2 of the OCVSD). Although not as customary, one approach is to perform real-time experiments, followed by fast-time. In reality, there is no set pattern on which validation technique should follow which, providing the selected one is appropriate. A thorough analysis actually involves multiple iterations using different types of evaluation tools. The "Banana Model" provides a good basic illustration of this process, but it presents the illusion that it is a one-way flow. In fact, there are two-way flows to every node in the model. With each iteration, the researcher uses information gained from the previous analysis, revisits assumptions and inputs, refines approaches, and selects the appropriate tool for future analysis. When considering the coordination of fast-time and real-time validation techniques, the link between them should reflect the iterative nature of the relationship, as demonstrated in Figure 1. This iterative approach suggests the need for greater coordination in the design, development and use of scenario. It allows for a common structure that supports data sharing and re-using information where possible. It is the aim of the best practices presented in this document to promote this coordination.

Fast-time

Real-time

Traditional Approach

Real-time

Fast-time

Alternative Approach

Fast-time

Real-time

Reality

Figure 1 - Iterative nature of fast- and real-time analyses.

6.1

Data Sharing and Re-Use between Fast-Time & Real-Time

Data sharing between fast- and real-time simulations can occur in two forms: one focuses on the data input, and one focuses on data output. The first type of data sharing concerns data originating from a single source that can be used as input into the scenario design for both fast- and real-time simulations. The benefits

B-12

of using common data sources are faster scenario development times, common baselines, and the ability to compare results. The second form of data sharing concerns the re-use of data used or produced by another/previous simulation. Such data could include the following: •

System performance, user response times, variability data

•

Human error models

•

Trajectory prediction inaccuracy, aircraft position inaccuracy

•

Event or series of events that alter the flow of aircraft

•

Traffic forecast for target years

•

Baseline measures (in terms of traffic, airspace, procedures, roles and responsibilities, operators, etc.)

The researcher should always use caution when considering sharing data in either form. He/she must ask questions: Are the chosen inputs and metrics suitable in both types of simulations? Is the specific type of data produced in the real-time simulation (e.g., controller response times) realistic enough to put into the fast-time model? What impact would a learning effect demonstrated in a real-time simulation data have on the fast-time model? Could the fast-time model erroneously magnify or overstate any of the impacts demonstrated in the real-time? To combat these risks, this document proposes the following best practices. Certain software packages are already under development with the aim of combining the characteristics of the fast-time and real-time simulation techniques. This development highlights the need for greater coordination in scenario development and the need for a common structure and leads to the first Best Practice recommendation in this document. bp 2-1. Communicate with fellow fast-time and real-time researchers In the interest of better coordination, those working on real-time simulations and those working on fast-time simulations should communicate and interact with each other regarding their work. Often the two parties are working independently on the same or similar issues. However, they should communicate about their efforts and share information on such issues as assumptions, results, and lessons learned. bp 2-2. Clearly state and document your design assumptions One of the difficulties in coordinating fast and real-time simulation activities and sharing data is the different types of assumptions that go into each, which can cause problems when trying to share or re-use data. As such, it is very important that both fast and real-time researchers clearly state their assumptions before performing their analyses. Perhaps a common template could be developed. That way any researcher could look at the template, and identify and understand the assumptions. This is vital for interpretation. bp 2-3. Clearly state the likely implications of your assumptions The researcher should state not only what the design assumptions are, but also what the implications of the assumptions are. This helps in the correct interpretation of a model, and

B-13

is particularly important if the assumptions are implicit. Documentation of the assumptions and implications is recommended to provide traceability by showing which outputs came from which sources, what assumptions affected what outputs, etc. bp 2-4. Standardize scenarios for use in both fast- and real-time simulations when possible To help facilitate coordination between scenario development in fast-time and real-time simulations, a standardized scenario development process and characterization scheme could be developed. These could be designed around each technique with common processes and areas noted for coordination purposes. This would be useful for designing scenarios, building scenarios, and possibly even comparing results after the simulations.

6.2

Technology Advancement

Due to the different objectives of fast-time and real-time simulations and the differing techniques and expertise involved for each coordination historically between fast-time and real-time has not been easy. It can be difficult to model certain concepts in the fast-time environment due to limitations in computer processing speeds, computer memory, or the fast-time models themselves. For example, fast-time simulation is weak when modelling human performance, such as in the case of Collaborative Decision Making (CDM). A view has been taken that some concepts just are not suitable for evaluation in the fast-time environment. Part of the problem has been that the fast-time simulation technology has not evolved to a level suitable to simulate the concepts. Emerging technologies, however, may solve this problem. Fast-time simulation software is continually under development and may in the future be used increasingly for what were traditionally subjects for real-time assessments only. Researchers may see more hybrid fast-time/real-time models in the years to come.

7. SCENARIO DEVELOPMENT CONSIDERATIONS Table 2 presents a list of recommended best practices to be considered when developing and designing scenarios. Each best practice is described in more detail following the table.

B-14

Table 2 - Best Practices for Scenario Development and Design 7 bp 2-5 bp 2-6 bp 2-7 bp 2-8 bp 2-9 bp 2-10 bp 2-11 bp 2-12 bp 2-13 bp 2-14 bp 2-15 bp 2-16

enario Development Considerations ow the research questions to drive scenario development p out and prioritize all potential scenario characteristics entify and maintain common scenario characteristics for comparison eck that the test environment can support your scenarios and be prepared to make compromises art with a baseline derived from current information (including traffic) nsider options such as “generic airspace” ek subject matter expertise during the development of your scenarios tain involvement from all stakeholders during scenario design fine and maintain necessary levels of realism ike a balance in the order of the scenario presentation enarios should carefully isolate key technologies that require large capital investments in order to be clear on potential benefits common list of scenario characteristics should be formed

Researchers have many factors to consider when designing scenarios. They know that carefully designed scenarios produce the most useful test results and that poorly crafted scenarios can preclude the achievement of test objectives. As such, the following section provides guidance on points to consider in the scenario development process. bp 2-5. Allow the research questions to drive scenario development While the researcher should strive to develop scenarios that model the operational environment, it is crucial to build scenarios to address the research questions and objectives. All perspectives (researcher, operational, and management) should be taken into consideration. This Best Practice is taken from Appendix 1 (bp 1-16). For more information, please refer to the text in that appendix. bp 2-6. Map out and prioritize all potential scenario characteristics Scenarios can be characterized on several different levels; by concept aspects (roles, procedures, and sequence of tasks), environment aspects (airspace, route structure, traffic volume), and event aspects (normal events, non-normal events). The researcher should develop a scheme to map out and prioritize potential scenario characteristics to aid in the development process. bp 2-7. Identify and maintain common scenario characteristics for comparison One of the biggest challenges in scenario development is to create scenarios that are experimentally comparable, but that are different enough to present a “new” problem. Experimenters often have to design scenarios that are similar (in complexity) but not exactly

B-15

same. Similar scenarios are an essential part of a good experimental design as they help with statistical comparisons. However, in real-time simulation, scenarios that are similar could also produce undesirable learning effects. The learning effects could negate the effects of the experimental conditions. Simply changing the aircraft call signs is not always adequate since controllers can often recognize similar air traffic patterns. Learning effects can be controlled somewhat by randomizing the order of experimental conditions. This Best Practice is taken from Appendix 1 (bp 1-17). For more information, please refer to the text in that appendix. bp 2-8. Check that the test environment can support your scenarios and be prepared to make compromises In keeping with the FAA / EUROCONTROL principles of concept levels of maturity (V1 to V5), scenarios should become more mature and rigorous as the concept becomes more mature. If the scenario changes, ensure that the simulation environment is capable of supporting the scenario requirements and can deliver the data needed to feed the selected performance metrics. bp 2-9. Start with a baseline derived from current information (including traffic) Baseline traffic samples should be derived from current traffic data. The use of baselines developed from past information should be avoided. Procedures, tools, and traffic schedules continuously change; therefore obtaining current information helps establish a more realistic baseline. There are some exceptions to this best practice in the fast-time environment since one has the ability to create a future baseline based on known future improvements. For example, if a runway is to be in place at an airport a year from now, modelling the current airport will have no value in comparing any new improvements. bp 2-10. Consider options such as “generic airspace” This Best Practice is taken from Appendix 1 (bp 1-15). For more information, please refer to the text in that appendix. bp 2-11. Seek subject matter expertise during the development of your scenarios The researcher should acquire subject matter expertise from air traffic control specialists, pilots, operational personnel, and/or individuals familiar with the domain of interest and corresponding traffic characteristics to assist in the development of scenarios. SMEs can help build and validate the scenarios. For example, they can assist the researcher in determining the number of scenarios required to achieve the desired simulation results. They can also point out weaknesses in the scenarios that may lead to incomplete data or missed opportunities. Having an SME participate in the scenario development process will add credibility and fidelity to any simulation effort. bp 2-12. Obtain involvement from all stakeholders during scenario design The researcher should try to obtain input from the various stakeholders involved with a simulation. Sponsors and potential users of the operational concept under test, airport and

B-16

airline representatives, and when possible, Airline Operation Center personnel, are key to gaining buy-in of simulation results. This best practice not only applies to scenario development, but to the entire simulation effort. bp 2-13. Define and maintain necessary levels of realism

While developing scenarios, researchers are usually interested in maintaining high realism related to aircraft mix, traffic density, sector geometry, routes, and procedures. Many times, the scenario development process starts with the collection of actual operational flight plan data, which provides the realism. The operational data with realistic traffic density and aircraft types works well for near-term initiatives. However, for future concepts, operational data may need considerable modifications in order to satisfy the experimental objectives. In such situations, a number of variables, such as future traffic load, future aircraft mix, and technologies may need to be modified. This Best Practice is taken from Appendix 1 (bp 1-19). For more information, please refer to the text in that appendix. bp 2-14. Strike a balance in the order of the scenario presentation In real-time simulation sound experimental design calls for randomization of scenario presentation, however, over-randomization can lead to an unfair variance in the results. Researchers should keep in mind that controllers are accustomed to a certain ‘order’ in their work. Over-randomizing the scenarios is unnatural. On the other hand, presenting scenarios in too similar of an order can lead to over-familiarization, or a learning effect. Balance is required. This is a problem in real-time, but not in fast-time, as a computer will not ‘learn’ the sample. bp 2-15. Scenarios should carefully isolate key technologies that require large capital investments in order to be clear on potential benefits When developing scenarios that require an assessment of multiple concepts, it is essential that you isolate the effects and performance changes of each concept, thereby making clear the benefits of each. There are certain concepts that require a large investment in time and resources, therefore the cost/benefit analysis must show the benefit of these particular concepts without the additional or cumulative affects of any other concept in your scenario. bp 2-16. A common list of scenario characteristics should be formed Most concept validation simulations are performed based on a single set of scenario conditions and does not take into account many aspects of the Air Traffic Control (ATC) environment. A solution to this problem would be to develop a common set of scenario characteristics based on the type of simulation performed. For example, at a minimum you should be required to perform one bad weather scenario, one runway change, one go around, and one level bust. Granted, this solution does not always fit; time, resource, and cost can significantly influence how many scenarios can be run.

B-17

Regardless, the use of such a list will enable some cross-referencing among different concepts or simulations. It is also particularly useful when assessing concepts of higher maturity by way of providing a set of 'bench-mark' tests.

8. TRAFFIC BUILDING Table 3 presents a list of recommended best practices to be considered when building traffic for scenarios. Each best practice is described in more detail following the table. Table 3 - Best Practices for Building Traffic 8 bp 2-17 bp 2-18 bp 2-19 bp 2-20 bp 2-21 bp 2-22 bp 2-23

lding Traffic arting and/or ending scenarios slowly is not usually efficient affic peaks and troughs may be relevant to research lidate aircraft performance n't exceed the airport projections when projecting enroute or terminal growth nsider city pairs, when possible anslate yearly forecasts (peak) into daily and hourly operations y to identify aircraft types and mix

Developing scenarios usually involves a large investment in time and cost. The majority of time spent developing scenarios is in building the traffic sample. A 'basic' traffic sample is not so hard to develop, but a large percentage of time is spent on ensuring the appropriate realism of the sample. For fast-time studies, about 20-30% of the total effort is spent on traffic preparation. bp 2-17. Starting and/or ending scenarios slowly is not usually efficient In real-time simulation, a common way of building a scenario is to initiate traffic gradually into the problem, allowing the controllers to ease themselves into the scenario. When the main part of the scenario is over, traffic usually tapers off. For an hour-long scenario, as much as 15 minutes on either end of the scenario might be dedicated to “ramp up/down” time. Given the time and cost restraints to run a simulation, this is not the most effective or efficient use of scenario time. A more efficient means of building a scenario is to begin with a normal traffic load. Depending on the study, practitioners may also end the scenario in the middle of a conflict or other problem. This Best Practice is taken from Appendix 1 (bp 1-18). For more information, please refer to the text in that appendix. bp 2-18. Traffic peaks and troughs may be relevant to research In real-time simulation, HITL practitioners often try to impose high workload levels on simulation participants to stretch the limits of the participant’s abilities and to test the limits of new concepts and procedures on the air traffic system. To do this, they typically design traffic scenarios to represent peaks or high levels of traffic activity. Slower-manifesting problems are generally avoided since the practitioner wants to make the best use of valuable time. Prior simulation research has shown, however, that operational errors often occur in the beginning of troughs, or lower levels of traffic, immediately following very high levels of traffic activity (this results because of a temporarily perceived reduction of

B-18

complexity and thus lower vigilance). In order to better emulate the operational environment and capture all possible conditions for human error, HITL practitioners should script a range of traffic activity into their scenarios; those which include both peak levels of traffic and troughs. This Best Practice is taken from Appendix 1 (bp 1-20). For more information, please refer to the text in that appendix. bp 2-19. Validate aircraft performance In real-time simulation, the validation of aircraft performance is very important because in general there is a greater need for realism. This is especially true if the purpose of the study is to assess the impact of a new aircraft type. Realistic aircraft performance is a key element in helping the participants to work in their normal manner, although there may be occasions where a lower level of realism can be acceptable. It is recommended that during scenario development SMEs (i.e., controllers) observe the aircraft movements to validate their realism. In fast-time simulation, the required degree of realism will depend on the purpose of the study. Queuing models may need nothing more than aircraft speed, while algorithms that are more complex are needed for conflict prediction and resolution. The standardization of aircraft performance in trajectory prediction is being investigated by other organizations; however, realism of aircraft performance continues to be an issue. bp 2-20. Do not exceed the airport projections when projecting en route or terminal growth When developing traffic samples, do not exceed the level of traffic that is projected for the origin or destination airports. Exceeding the level of traffic at the origin or destination airport will cause unrealistic delays and throughput in the airspace system. For example, do not have 80 aircraft per hour flying between airports with a 35 aircraft per hour capacity. bp 2-21. Consider city pairs, when possible A common practice in today's environment is to simulate each airport or section of airspace as a stand-alone system, without taking into account the affects on certain city pairs. When generating traffic samples, the origin and destination airports must be able to handle the level of traffic that is projected for each. Traffic should be adjusted based on terminal area forecasts and/or other business models, to provide a realistic future traffic sample. Certain models or simulators require city pairs to execute properly, so having city pair information will aid in re-use and/or real-time/fast-time coordination. Finally, traffic generated for a certain city could be used in metro area analysis. bp 2-22. Translate yearly forecasts (peak) into daily and hourly operations For most simulations, yearly forecasts must be translated into daily and hourly operations. To do this, a baseline of current traffic should be developed through data gathering and adjusted based on traffic forecast. These techniques need examination since they have not been perfected or standardized, especially when it comes to smoothing peak periods of traffic.

B-19

A point of caution here is to take abnormal peaks or troughs into account. For various reasons traffic levels may experience an abnormal growth at certain times (e.g., an annual major event in the area). Such occasions, although not normal, should be considered. bp 2-23. Try to identify aircraft types and mix Environmental studies require the analysis of specific aircraft types especially during noise studies, so identifying individual aircraft types and establishing a fleet mix is essential. Some simulations only require aircraft groups or classes, but when the traffic sample is used in other simulations, it becomes very difficult to re-engineer without specific knowledge of aircraft types and composition of fleet mix. Identifying individual aircraft types also helps in coordination between fast and real-time simulations.

9. SPECIFIC EVENT SCRIPTING Table 4 presents a list of recommended best practices to be considered when scripting events. Each best practice is described in more detail following the table. Table 4 - Best Practices for Scripting Specific Events 9 bp 2-24 bp 2-25 bp 2-26 bp 2-27 bp 2-28

ipting Specific Events an more scripted events than you need to ensure desired results epare participants as necessary not try to script operational errors nsider scripting system-wide events nsider the use of dynamic scripting

A scripted scenario is one that deliberately introduces a disturbance into the system. Such a disturbance is generally known in advance to the validation team but not the simulation participants. Scripted scenarios are particularly useful for assessing the impact of 'unplanned' events in a simulation environment where if you did not deliberately introduce them, you would probably not encounter these (often critical) situations. bp 2-24. Plan more scripted events than you need to ensure desired results When accessing a certain operational concept, you may have certain events happen to show the benefits of the concept. In this situation, you should plan more scripted events than needed to ensure desired results. For example, script nine potential conflicts to obtain at least four. However, adding more events should not compromise realism. bp 2-25. Prepare participants as necessary In real-time simulation, it is usually advisable to prepare participants about potential events they may encounter in the simulation. However, any pre-warning should be kept to a minimum, since you still require controllers to work normally and not in a contrived manner. For example, consider the case where a simulation is investigating a future concept, such as the effects of a catastrophic failure of the Global Positioning Satellite (GPS) system when it is used as a sole source of navigation. Because controllers to date have never used GPS as a sole source of navigation, pre-warning of planned simulation events related to its failing

B-20

is necessary so that controllers are not completely unfamiliar with the situation and what procedures to implement when such an event occurs. bp 2-26. Do not try to script operational errors In real-time simulation, it is useful to study the occurrence and impact of operational errors. There may be a temptation to script certain events to induce such errors. However, this practice is not recommended. Errors such as these should simply be allowed to happen in order not to bias any results or participant opinions of the concept. Should such errors occur, be careful how you report them. The report should not be of a 'telling tales' nature. Perhaps rather than referring to them as operational errors, the researcher could introduce some adaptation of 'performance scoring'. This approach should enable the same discussion but it appears less negative on the participants. It is also necessary because in any simulation you may get an unfair proportion of operational errors through controller learning or experimenting with the system. bp 2-27. Consider scripting system-wide events As well as using scripts that relate to single events in the system being simulated, consider introducing system-wide scripts such as airport closures and system failures. Obviously this could involve considerable planning and may only be done once or twice in a simulation. However, stakeholders are increasingly concerned about the affects of such system-wide events. bp 2-28. Consider the use of dynamic scripting In real-time simulation, predetermining scripts may not always be easy, especially when evaluating new systems or if the researcher is unsure of the events beforehand. The dynamic nature of the system means that events can unfold rather differently than planned. A way to adjust when scripts do not go as planned is to use dynamic scripts. For example, consider the example given in Section 3.2.1, which listed particular aircraft expected to be involved in a particular sequence of events. If executing this dynamically, instead of picking a particular aircraft, wait and see how the situation unfolds and pick another aircraft. However, it should be noted that this dynamic nature also means it can be difficult to accurately reproduce events or compare results between replications. Therefore, it is not recommended in experiments where performance measurements comparisons will be made. In the early stages of concept development, allowing scenarios to unfold dynamically may help in the development of roles and procedures should these not already be defined.

10. VALIDATION DATA REPOSITORY 10.1 How the VDR Can Help The Validation Data Repository (VDR) is a means of managing validation data. It is designed to support the development of a common reference framework with which validation activities can and should be mapped. With regard to ‘scenarios,’ this has already begun with the introduction of an established information structure that provides:

B-21

•

Summary information about a scenario (e.g., name, acronym, version, summary, ATM environment description, dates, type)

•

Descriptions of relevant operational concepts under test

•

Defining characteristics of the scenario (e.g., geographic area, airspace design, traffic samples and profiles, system configurations, task allocations, procedures, event sequences)

•

Hyperlinks to detailed source reference documents

The VDR allows different types of scenario to be recorded in a common format, for example, target scenarios, baseline/reference scenarios as well as the test/validation scenarios. In terms of the re-use of information, the VDR enables mapping of scenarios to exercises, or in other words, provides a means of referencing scenarios. The VDR can also map a scenario to more than one exercise. This means standard scenarios can be defined and linked to many exercises. These exercises may be of different types, for example, fast- and real- time simulations, flight trials, safety case assessments, and economic appraisals. Information regarding the defining characteristics of a scenario can also be re-used. A scenario may have one or more scenario parameters (the detailed defining characteristics). Scenario parameters may be applicable to more than one scenario, for example, those dealing with airspace characteristics and traffic characteristics. Such parameters need only be defined once within the VDR because they can be made available for linking to many scenarios.

10.2 How the VDR Treats Scenarios The VDR provides a structure that defines information in a consistent manner to enable scenario comparisons, but also provides sufficient flexibility to allow researchers to make comparisons based on their needs. Figure 3 depicts the information structure relating to scenarios in the VDR.

B-22

OPERATIONAL CONTEXT OF VALIDATION SPECIFIC OPERATIONAL CONCEPTS One or more Concepts

e.g. ADS-B Application Package 1 ASAS Crossing Free Flight

VALIDATION PROJECT STRUCTURE PROJECTS

One or more Scenarios

SCENARIOS

One or more Exercises One or more Scenarios

EXERCISES

One or more Scenarios

REFERENCE DOCUMENTS

One or more Parameters SCENARIO PARAMETERS

e.g. Traffic Sample & Characteristics Airspace Design CNS configuration Avionics etc.

Figure 2 - The role of scenarios in the VDR.

11. SUMMARY OF BEST PRACTICES IN SCENARIO DESIGN The collaborative Action Plan 5/Action Plan 9 Workshop brought together a range of practitioners from the U.S. and Europe with experiences in fast-time and real-time simulation to discuss scenario development for purposes of concept validation. The following is a summary of those discussions and the best practices that emerged related to scenario design. The practitioners’ expertise and backgrounds varied; therefore, a common definition of scenario had to first be developed. To satisfy both fast-time and real-time practitioners, a definition of scenario was created for each technique. Scenario uses and scenario objectives were presented, providing a common ground for topic discussions. Coordination of fast and real-time simulation was a topic that generated much discussion. It was clear that few practitioners had coordinated fast-time and real-time activities. It was also clear that many practitioners of fast-time and real-time simulation were inexperienced in each others’ areas of expertise. Increased communication between fast-time and real-time practitioners became one of the first best practices. Standardizing scenarios and clearly stating and documenting design assumptions and the implications of such assumptions for both real-time and fast-time was recommended to allow for data sharing and re-use between the two techniques. Through topic and informal discussion, it was clear that the issue of coordination between fast-time and real-time activities should be addressed further. Although there are quite a number of factors to consider, the practitioners composed the 28 best practices found in this appendix for scenario development. Building traffic takes a considerable amount of effort in scenario development. As such, the practitioners identified key considerations related to this activity. For example, with respect to traffic volume in real-time simulation, starting or ending scenarios slowly is not usually B-23

efficient and traffic peaks and troughs may be relevant to research. In the case of aircraft performance validation, unrealistic performance can be detected by an experienced controller. The practitioners also addressed building traffic to represent growth based on airport and city pair projections, translating yearly forecasts into daily and hourly operations, and trying to identify the appropriate aircraft types and mixes. Scripting an event is deliberately introducing a disturbance into the system. In real-time simulation, this would be accomplished by planning a specific type of event and introducing it during the simulation run to induce some kind of human response. In fast-time simulation, a scripted event is usually introduced at a specific time during a simulation run. The best practices resulting from the workshop that relate to scripted events mostly deal with realtime simulation. Best practices include the following: Planning more scripted events than needed ensures desired results. Preparing participants as to what to expect is a common practice, but one should keep this to a minimum. Intentional scripting of operational errors should not occur, operational errors should happen naturally in the simulation. Consideration should be taken to script system wide events. Practitioners should keep in mind that the VDR is a valuable tool that provides a means of storing, sharing and disseminating data, including scenario information, and it establishes a common reference framework with which validation activities can and should be mapped. The VDR provides an established information structure for managing scenarios. Based on feedback surveys, most practitioners felt that the workshop was useful and that other workshops should be developed in the future. Metrics and measures, reporting of results, and more specific scenario development topics (separate fast and real-time) were some of the suggestions for future workshop topics. Bringing together practitioners from different backgrounds to discuss "best practices" was a challenge, but overall the experience was a successful one.

B-24


Appendix C Scenario Use in Concept Validation

Prepared By: Nigel Makins EUROCONTROL Experimental Centre Albert Schwartz, Karen Buondonno, Sherri Magyarits Federal Aviation Administration William J. Hughes Technical Center Manuel Dorado Aeropuertos Españoles y Navegación Aérea

February 17, 2006

1

Acknowledgments The persons listed below attended the November 2004 workshop in San Jose, California. This paper is a collection of their ideas, experience, and expertise. The authors would like to thank the attendees for their dedicated involvement and contributions to this effort. The authors would also like to recognize and thank Kristina Burch (FAA William J. Hughes Technical Center) for her contributions to this report.

AENA, Spain:, José Miguel de Pablo, Manuel Dorado ATAC: David Holl CSSI, Washington, DC: Stephane Mondoloni EUROCONTROL Experimental Centre, France: Robin Deransy, Nigel Makins EUROCONTROL HQ, Belgium: Ulrich Borkenhagen, Gerald Mcauley, Hans Wagemans FAA CAMI: Carol Manning FAA Headquarters, Washington D.C.: Diana Liang FAA William J. Hughes Technical Center, NJ: Sherri Magyarits, Albert Schwartz INECO: Victor Bustos MITRE: Dave Millner NASA Ames Research Center: Parimal Kopardekar, Sandy Lozito, Lynne Martin, Tom Romer, Savvy Verma NLR, Netherlands: Jos Beers, Juergen Teutsch Seagull: Dave Schleicher San Jose State University, California: Kevin Corker

C-2

Table of Contents 1. Introduction ...............................................................................................................6 1.1

Background ..................................................................................................6

1.2

Objectives.....................................................................................................7

1.3

Definitions.....................................................................................................7

2. Method ......................................................................................................................7 2.1

Web Discussion Group .................................................................................7

2.2

Workshop Format .........................................................................................8

2.3

Document Structure......................................................................................9

3. Overview of the AP5 Maturity Model..........................................................................9 3.1

Milestone V1 - Project scope ......................................................................10

3.2

Milestone V2 - Feasibility ............................................................................11

3.3

Milestone V3 - Performance capabilities .....................................................11

4. Workshop Outcomes ...............................................................................................11 4.1

Identification of the Concept Comparability Issue........................................12

4.2

Addressing Concept Comparability .............................................................12

4.3

Sub-question 1 - Scenarios and Maturity Model..........................................13

4.4

Sub-question 2 - Scenario Components .....................................................15

4.5

Sub-question 3 - ATM Objective Scenarios.................................................19

4.6

Sub-question 4 - Validation Technique Scenarios.......................................19

4.7

Sub-question 5 - Feasibility of Standard Scenarios.....................................20

5. CARE/ASAS Case Study.........................................................................................22 6. Scenario Guidelines ................................................................................................23 7. Conclusions and Recommendations .......................................................................24 References....................................................................................................................27 Attachments C1 - Scenario Guidelines

C-3

List of Tables and Figures Table 1. Core Components ........................................................................................................18 Table 2. Baseline Scenario Input and Output Requirements .....................................................21 Figure 1. The Action Plan 5 Maturity Model ...............................................................................10 Figure 2. Progression of a Scenario ..........................................................................................14

C-4

Executive Summary Action Plan 5 (AP5-Validation and Verification Strategy) and Action Plan 9 (AP9-Air Traffic Modeling of Operational Concepts) conducted the Third Federal Aviation Administration/EUROCONTROL Practitioners’ Workshop, November 2-4, 2004. The purpose of the workshop was to discuss the development of simulation scenarios for validation exercises. The workshop was attended by 10 European and 14 United States practitioners, all of whom were experienced in fast-time and real-time simulation for concept validation. The workshop objective was to answer the following three questions and establish guidelines for developing scenarios for concept validation: • • •

What would ‘typical’ scenarios cover (performance, behaviors) at each level of development? What would be the components of a scenario at different maturity levels? How can the ability to compare results across projects and concepts be fostered (i.e. what aspects of a scenario can be standardized)?

To help attain the objective, five sub-questions were developed: 1. Can the Action Plan 5 (AP5) Maturity Model be used as a framework for developing ‘typical’ scenarios? If so, what sort of issues (performance, behaviors) would those scenarios expect to cover at each stage? 2. What are the components of a typical scenario? How do they differ at each stage? 3. Are air traffic management (ATM) objective-specific scenarios needed (i.e. for capacity, safety, environment etc.) and how should these be linked? 4. Are validation technique-specific scenarios needed (e.g. fast-time or real-time simulation) and how should these be linked? 5. In what conditions would the use of standard scenarios be unfeasible or inappropriate? During the workshop, practitioners introduced each of the sub-questions by providing a short briefing followed by a discussion of each session. Practitioners created three breakout groups to discuss scenario components and the definition of a ‘typical’ scenario. They presented their consolidated recommendations to all workshop participants for final discussion and consensus. As in the previous appendices, this appendix documents the results of the workshop. However, the topics discussed during this workshop did not generate specific best practice recommendations as in the previous workshops. Instead, this appendix includes general recommendations on scenario development and the application of the Maturity Model. It presents guidelines for simulation scenario development and validation as a supplement for experienced practitioners who perform concept validation activities.

C-5

1. INTRODUCTION The Federal Aviation Administration (FAA) and EUROCONTROL organized the third Action Plan 5 (AP5) workshop in conjunction with Action Plan 9 (AP9). The workshop took place in November 2004 at the San Jose State University in San Jose, California. The third workshop participants included 14 practitioners from the United States (U.S.) and 10 practitioners from Europe representing both fast-time and real-time simulation fields. The U.S. participants included researchers from the ATAC, FAA William J. Hughes Technical Center, FAA Headquarters, FAA Civil Aerospace Medical Institute (CAMI), National Aeronautics and Space Administration (NASA) Ames Research Center, MITRE, CSSI, Seagull, and San Jose State University. The participants from Europe included researchers from Aeropuertos Españoles y Navegación Aérea (AENA), EUROCONTROL Headquarters, EUROCONTROL Experimental Centre, INECO, and Nationaal Lucht en Ruimtevaartlaboratorium (NLR).

1.1

Background

The FAA/EUROCONTROL Coordination Committee was established in December 1995 during the Second FAA/EUROCONTROL Research and Development (R&D) Symposium, held in Denver, Colorado. The focus of the FAA/EUROCONTROL Coordination Committee was to define priorities in terms of common actions and agendas of both organizations. The Committee identified areas of mutual interest where the FAA and EUROCONTROL could work together in R&D and defined several R&D cooperative tasks, which are referred to as ‘Action Plans’. The goal of AP5 is to determine a unified strategy for validating and verifying the performance, reliability, and safety of Air Traffic Management (ATM) systems. One objective of AP9 is to promote a mutual understanding between the U.S. and Europe on the use and development of fast-time simulation models for modelling of air traffic operational concepts. An objective shared by AP5 and AP9 is to develop detailed best practices for performing tasks associated with the verification and validation of ATM systems. The development and subsequent use of these best practices will allow for sharing of information and comparison of results among all researchers. Best practices covering the use of metrics and measures, data collection, data analysis, and reporting are useful for each type of validation exercise, as they take into account several factors including resources, cost, and most importantly, prior ‘lessons learned.’ In order to capture lessons learned from a range of research organizations, several workshops were conducted. The attendees of these workshops included experienced practitioners in their respective fields and in the topics presented at the workshops. This appendix documents the results from the Third FAA/EUROCONTORL Practitioners’ Workshop, where the topic for discussion was the development of simulation scenarios for validation activities. The discussions from this workshop did not generate specific best practice recommendations for practitioners as in previous workshops. Instead, this appendix includes general recommendations on scenario development and the application of the Maturity Model. It presents guidelines for simulation scenario development and validation as a supplement for experienced practitioners who perform concept validation activities. More workshops are planned to address other pertinent topics.

C-6

1.2

Objectives

The workshop objective was to discuss the topic of standardization and comparability of scenarios at all levels of concept development and to establish guidelines for developing scenarios for concept validation.

1.3

Definitions

The following definitions of Validation Scenario and Scenario Components were used by the workshop participants and are provided as a reference for this document. 1.3.1 Validation Scenario A validation scenario is an extension of the operational concept scenario. It is a representation of the events, actors, and interactions of the operational scenario applied in a simulation environment. The objective is to excite the performance and interactions described or expected in the operational scenarios. The simulation environment refers to various configurations of airspace, traffic samples, weather, failure modes, and any other controllable variables that might affect the performance of the ATM system. In this way, a validation scenario will test the assumptions in the concept scenarios and thus the concept design. The validation scenario is that which is actually implemented in a simulation in order to provide robust feedback in terms of whether or not a proposed concept can be implemented in the operational environment. Validation scenarios are actually run in the experiments, therefore they will require specific traffic files, airspace files, and possibly scripted events in order to execute properly. They will also have associated validation objectives and be able to address specific metrics and indicators. 1.3.2 Scenario Components or Parameters The Validation Data Repository (VDR) User Guide [Validation Data Repository: User Guide] defines scenario parameters to be the set of parameters used to define, in detail, a given scenario used in the validation exercise either as the operational environment under test or the operational environment that the test is being compared with. Examples of scenario parameters include traffic sample, procedures, and system configuration.

2. METHOD 2.1

Web Discussion Group

Prior to the workshop, a web discussion group was created to introduce the main topics and to provide a central location for early discussion. Formats from past workshops included pre-selected topics, pre-selected speakers, and discussion of those topics at the workshop following speaker presentations. This format worked well, but only provided a few selected participants with insight into the topics before the workshop. The preliminary web discussion group was added to allow all of the practitioners (as well as experts who did not attend the workshop) the opportunity to interact and familiarize themselves with areas of discussion before the workshop began. A web discussion group is defined as a computer conference devoted to a specific subject, having an initiator, members, readers, and

C-7

possibly moderators. The web discussion was hosted by the Collaborative Discussion and Information Management Service (CDIMS). CDIMS provided administrative services and an established discussion group environment for the selected topic. The CDIMS administrator established five topic areas for discussion, which coincided with the five sub-questions (see Section 2.2), and allowed participants access to the system via a web link (i.e., https://callisto.cdims.act.faa.gov/Scenario/AP9S.nsf). Workshop organizers sent an invitation to over 75 practitioners’ from both Europe and the U.S. to participate in the discussion, however only a limited number participated. Despite the limited participation, the discussion did provide some insight into scenario development and the results of the discussion group are incorporated throughout this document.

2.2

Workshop Format

To address the workshop objective, the practitioners were asked the following question: What would ‘typical’ scenarios cover (performance, behaviors) at each level of development, what would be the components of a scenario at different maturity levels, and how can the ability to compare results across projects and concepts be fostered (i.e. what aspects of a scenario can be standardized)? Workshop organizers developed five sub-questions to adequately address the above question: 1. Can the AP5 Maturity Model be used as a framework for developing ‘typical’ scenarios? What sort of issues (performance, behaviors) would those scenarios expect to cover at each stage? 2. What are the components of a typical scenario? How do they differ at each stage? 3. Are ATM objective-specific scenarios needed (i.e. for capacity, safety, environment etc.) and how should these be linked? 4. Are validation technique-specific scenarios needed (e.g. fast-time or real-time simulation) and how should these be linked? 1 5. In what conditions would the use of standard scenarios be unfeasible or inappropriate? During the workshop, practitioners introduced each of the sub-questions by providing a short briefing, followed by a discussion session. Three breakout groups discussed scenario components and the definition of a ‘typical’ scenario. Results from each breakout group were then presented to all practitioners. The three breakout groups discussed the following scenario topics: 1. Breakout Group 1 - Identify the ‘typical’ fast-time simulation components in milestones V1-V3 of the Maturity Model. 2. Breakout Group 2 - Identify the ‘typical’ real-time simulation components in milestones V1-V3 of the Maturity Model. 3. Breakout Group 3 - Answer the following questions relating them to milestones V1V3 of the Maturity Model. 1

Question 4 was initially, “Are validation-tool-specific scenarios needed and how should these be linked?” It has been modified to reflect the nomenclature found in the VDR.

C-8

• • • •

2.3

In what conditions would the use of standard scenarios be unfeasible or inappropriate? Are there reasons for thinking that using standard scenarios in a validation methodology will have a negative impact? What are the risks? Are there clearly identifiable situations where standard scenarios should and should not be applied?

Document Structure

The format of this document follows the structure and flow of the workshop. •

Section 1 provides background information as well as the objectives of the workshop.

•

Section 2 describes how the workshop was conducted.

•

Section 3 discusses an overview of the Maturity Model as described in the Operational Concepts Validation Strategy Document (OCVSD) [Validation Data Repository: User Guide] developed by AP5. The Maturity Model is used to show five concept development milestones, from idea to implementation. This model was used to establish a framework for addressing the main question and sub-questions of the workshop.

•

Section 4 examines the overarching issue of concept comparability, explores the five sub-questions posed during the workshop, and incorporates the results of the workshop breakout groups.

• •

Section 5 presents a sample case for the use of the Co-operative Actions of R&D in EUROCONTROL (CARE)/Airborne Separation Assistance (ASAS) scenario template, Section 6 provides a guideline of scenario components for scenario development. This component list was developed based on breakout group feedback and the CARE/ASAS scenario templates [CARE/ASAS Activity 2: VF Project-WP1-D1].

•

Section 7 summarizes the overall conclusions and recommendations of the workshop.

3. OVERVIEW OF THE AP5 MATURITY MODEL The AP5 OCVSD proposes a five-level concept maturity scale ranging from V1 (idea) to V5 (implementation) (see Figure ) to guide validation objectives. The use of this model is also proposed in the ‘European - Operational Concept Validation Methodology’ (E-OCVM) [CARE/ASAS Activity 2: VF Project-WP1-D1]. The OCVM describes how the model is most applicable to concept investigation during the first three milestones of the Maturity Model (i.e. V1, V2, and V3). When the V4 (industrialisation, procedures approval) and V5 (implementation) milestones become the focus of the validation plan, more rigorous evaluation methodologies will be required.

C-9

ATM Concepts - Levels of Maturity IMPLEMENTED PROCEDURE/ PROCESS

IDEA

ep nc co s h le li s c i p a b ri n p

nc co of /s o f pe ro t y ‘ p to a l Pr o

pr

iti

st t

of s n re tio d u ta c e e n ro p em s / p l se Im es oc pr l n/ io v a a t ro lis p p r ia a s t re d u du In c e ro P n io at ns g r tio t e la in u p t s im ce s on op C e t’ ep

In

E

V1

V2

V3

V4

V5

Validation checkpoints - increasing rigour addressing specifics e.g. safety, human factors, technical, cost.

Figure 1. The Action Plan 5 Maturity Model The assumption behind this Maturity Model is that an ATM concept will take time to develop into an application and that the testing process should support this ‘maturing’ process. For example, as a concept matures, ‘programme managers’ should be aware of what should have been done and what can be expected in terms of results, in order to set appropriate programme and project objectives. This Maturity Model reflects the current state of thinking but requires further development to determine the type of experimentation and assessment that should be considered appropriate at each stage of development. The model should not be seen as a strict description of how ideas actually develop into applications because the reality is much more complex. However it is a framework that is intended to help understand what information should be available before an idea can be taken on by external stakeholders (e.g. industry) for further development. The first the milestones (V1-V3) were considered and referenced during the workshop and are therefore summarized in the following sections.

3.1

Milestone V1 - Project scope

The scope of the concept is generated at this milestone of the Maturity Model, with key aspects of the concept described at a high level. The emphasis should be on identifying the following: • • • • •

forecasted ATM system bottlenecks that the concept addresses in the envisaged timeframe, proposed concept and alternative approaches to resolving those bottlenecks, state of knowledge regarding the concept under development, potential costs centres - where costs will be found - ex. software, hardware, technology, staff training, increased fuel burn, etc. benefit mechanisms,

C-10

• •

3.2

stakeholder perspectives about the concept including specific issues that further work must address, and modelling requirements to evaluate concept.

Milestone V2 - Feasibility

The emphasis of this milestone should be on the development of the concept into an application through iterative prototyping and simulation. The objective of this stage is to identify the risks towards the proposed new concept and to address them in principle. This includes an understanding of the behavioural characteristics of the concept application and determining how to operate it safely and efficiently. At this stage, the emphasis should be on usability and safety issues with operational acceptability as a major aspect. Capacity, efficiency, and environmental evaluations can still lack stability while the usability and safety issues are being stabilised. Typical evaluations will cover: • • • • • • • •

operating procedures for controllers and pilots in nominal and non-nominal situations, phraseology - for nominal and non-nominal situations, impact on infrastructure, human-machine-interface (HMI), safety arguments, potential costs, potential benefits, and potential risks.

Milestone V2 is reached by having achieved confidence that all identified risks can be mitigated.

3.3

Milestone V3 - Performance capabilities

Having determined how to operate the concept application and having addressed all potential risks against the concept, the next milestone should be to concentrate on determining performance capabilities and therefore allow cost and benefit assessments. Typical evaluations will cover: • • • • •

safety, capacity (delay), applicability (airspace or airports where most/least appropriate), cost determination, and benefits assessments.

4. WORKSHOP OUTCOMES This section discusses an overarching issue concerning concept comparability that surfaced during the workshop, and expounds on each of the five sub-questions addressed by the participants. The practitioners only examined the first two sub-questions in detail. The information that follows is based on their responses. The information furnished for the remaining sub-questions is based on the collaborative authors’ expertise.

C-11

4.1

Identification of the Concept Comparability Issue

The workshop organizers generated five sub-questions to focus the efforts of the participants. An underlying issue that emerged during the workshop, namely, the ability to compare concepts at an early stage of development to determine where to focus limited development resources, linked these five questions. Budget allocation is a strategic management problem common to U.S. and European organizations concerned with formulating ATM development strategies. Validation exercises would be expected to provide information to support the decision making process in order to answer the following questions: “Which concepts show the greatest potential?” and “How long and complex is the implementation process likely to be?” Information on concept performance capabilities is required in order to formulate and review a strategy. It becomes very difficult for the decision makers to apply this information to emerging concepts to determine a reasonable development strategy if this information is based on ad-hoc, customized validation exercises. Comparability of results is also key to external stakeholders who need to make investment decisions. Thus, the use of standard scenarios would be an attempt to provide some comparability to the many validation results. Some advances are being made to provide standard indicators and metrics (i.e., INTEGRA), which may be most useful to the decision makers if they are used to measure performance in comparable air traffic scenarios. Unfortunately, a gap seems to exist between managements’ information needs about comparative performance and the validation practitioners’ appreciation of the need to customize validation exercises in order to measure specific performance issues. This gap (discussed further in Section 4.2) is highlighted by the changed dynamics of the workshop agenda. Originally developed to discuss the five sub-questions, the issue of concept comparability became the focal point of discussion. By the end of the workshop it was clear that the original questions were symptomatic of the fundamental problem.

4.2

Addressing Concept Comparability

During the workshop, the practitioners’ most prevalent comments related to the difficulty of using standardized scenarios in their validation experiments, citing that validation exercises usually require customization of airspace and traffic in order to focus on specific issues. Likewise, aspects of early concept development may also have a need to incorporate specific issues, such as dense en-route airspace, terminal airspace, or large airports. On the other hand, management has a strategic need to understand any differences that may exist regarding how potential concepts cope with specific issues so that they can determine which concepts can be pursued and how to best allocate budgets. The sub-questions posed during the workshop all touched on the underlying issue concerning the gap between the practitioners’ need for customization and managements’ strategic need for comparability. For example, sub-question 1 regarding the use of the concept Maturity Model challenged the practitioners to determine the stage of concept development at which standardized scenarios could be applied. The assumption was that customized scenarios would be needed in early stages of concept maturity, but standardized scenarios may be feasible in more mature concepts when procedures, phraseology, and interface issues become stable. The question forced the practitioners to consider the benefit of incorporating standard scenarios in the validation process.

C-12

However, without intention, sub-question 2 particularly highlighted the existence of the disparity on this subject. In fact, two of the three breakout groups chose to concentrate on ‘components of fast-and real-time simulations’ without addressing the ‘standardized or typical’ element of the question. This live example during our own workshop further emphasizes practitioners’ reluctance to consider the use of standardization in lieu of customized scenarios. Though many thought provoking issues were raised and some solutions are suggested in the sections that follow, more effort needs to be expended to close the gap on this matter between management and practitioners involved in the validation process.

4.3

Sub-question 1 - Scenarios and Maturity Model

The first question “Can the AP5 Maturity Model be used as a framework for developing ‘typical’ scenarios?” tries to identify a framework for developing ‘typical’ scenarios. This led to a secondary question, “What sort of issues (performance, behaviors) would those scenarios expect to cover at each stage of the Maturity Model?” In order to remain consistent with the OCVSD, the Maturity Model was selected as a possible framework. The term ‘typical’ scenario refers to a representative scenario such as an airspace scenario or airport scenario, but does not imply a ‘standard’ scenario. The description of specific scenarios or types of scenarios to be applied to the evaluation process through the different stages of development of the Maturity Model could help to ensure that suitable testing has been achieved at each stage. 4.3.1

Typical Scenario Characteristics - Milestones V1-V3

Due to time constraints there was limited discussion on this topic. However, typical scenario characteristics at the first three maturity levels are included in this section based on the authors’ experience. Additional discussion and research are recommended to provide further insight for this topic. Figure 2 illustrates the progression of a scenario for a specific program/project during the first three maturity levels.

C-13

Standardized Baseline Scenario

Scenario Guidelines

Level High Level cription Description Based Based enario Scenari

V1

Detailed Detailed Component Component Based Based Scenario Scenari

Concept Concept Developer* Developer*

Concept Concept eveloper/ Developer/ actitioner Practitioner

Scenario Scenari Review Revie

V2 Validation Technique Specific

More Detailed More Detailed Component More Detailed More Detailed Component More Detailed Based Component Component Based Component Scenari Based Based Scenari Based Scenari Scenario Scenario Scenari Scenario Scenario Reuse Reus

actitioner Practitioner

V3

oncept Concept eveloper/ Developer/ actitioner Practitioner

* To simplify the diagram Concept Developer could also be the Sponsor

Figure 2. Progression of a Scenario The first level of the progression chart starts with a high level description based scenario. This scenario type would be developed during the V1 milestone of the Maturity Model and would contain the scenario components as shown in Table 2 of Section 4.7. The high level description based scenario would be a less detailed description of the scenario and would be developed by the concept developer and/or sponsor. During stage V1, scenarios begin to scope the applicability, benefits, and stakeholder issues of the concept under examination. The scenarios demonstrate how the concept would operate in the envisaged timeframe, region, and area of operations (en-route, airport, etc.) with an appropriate traffic forecast. The scenarios should also support stakeholder appreciation of the concept in order to identify potential issues that the validation plan would have to address. Results should clearly illustrate the mechanisms that would generate the intended benefits. Scalability of these potential benefits is important at this stage, thus it would be necessary to generate macroscopic scenarios based on traffic forecasts of the whole of European Civil Aviation Conference (ECAC) or National Airspace System (NAS) to support evaluation of the expected benefits. From this high level scenario description, a progression to milestone V2 yields a more detailed component based scenario that would be developed through coordination between the practitioner and the concept developer to ensure a proper interpretation of the scenario. Section 6, entitled “Scenario Guidelines” provides a description of the level of detail required for scenario development at this stage. Once the detailed component based scenario is created a more detailed validation technique specific scenario can be generated. Validation is required in the V2 milestone of the Maturity Model and is the responsibility of the practitioner who has the working knowledge of particular validation techniques (e.g., a fasttime airport model). To ensure that the more detailed scenarios match the concept developer’s vision, a review loop is established between the practitioner and concept

C-14

developer. Once complete, the detailed component and validation technique-specific scenarios can be reused in later stages of the Maturity Model. A V2 milestone scenario encompasses the development of the concept to a state where its feasibility, behavioural characteristics, and operating procedures in all situations (nominal and non-nominal) are stable and adequately described. Operating procedures included in these scenarios should include ‘safety situations’ whereby specific identified hazards are introduced in order to ensure that operating procedures can cope with these types of events. At this stage the scenarios should encompass more than one geographical location to help ensure that expected benefits are not constrained by peculiar local conditions. The scenarios should also clearly demonstrate the operational demands on the enabling technologies to ensure that technical requirements are well founded. The V3 milestone determines the performance capabilities of a concept once stable operating procedures have been established. Scenarios should be used to determine potential capacity and delay through use of typical forecasted high demand scenarios.

4.4

Sub-question 2 - Scenario Components

In Section 1.3, definitions for validation scenario and scenario component were provided. Two breakout groups were formed during the workshop to provide details as to what components comprise a validation scenario. The first group was tasked to identify the “typical” components from a fast-time simulation perspective while the second group was tasked to identify the “typical” components from a real-time simulation perspective. As a starting point, the following component list was provided to both breakout groups: • • • • • • • • • • • • 4.4.1

Traffic (Schedule, Flight Plans, Fleet Mix, Origin/Destination Pairs, Timeframe, Forecast, etc.) Airspace Characteristics (Sectors, Route Structure) Airport Characteristics Operational Procedures Environment (Weather, Noise, Emissions) Traffic Flow Management Scripted Events (System Failures) Human Factors Technology Configuration (Communication/Navigation/Surveillance) Other (Safety Situations, Security Situations) Aircraft Performance Assumptions and Constraints Breakout Group 1 - “Typical” Fast-Time Simulation Components

The goal of Breakout Group 1 was to identify ‘typical’ fast-time simulation components for milestones V1-V3 of the Maturity Model. The group determined that at each level of maturity the components remain constant, but the information contained in the component varies. This is due to the fact that as the maturity level increases there is a need for higher fidelity simulation (fast- or real-time) to validate and test the rigor of the concept. Breakout Group 1 identified five core components; four came from the original list and one additional component, called output metrics, was added.

C-15

• • • • • •

Core components for fast-time simulation: Traffic (Schedule, Flight Plans, Fleet Mix, Origin/Destination Pairs, Timeframe, Forecast, etc.) Airport Characteristics Airspace Characteristics Operational Procedures Output Metrics

While there may be some debate as to whether metrics are a component of a scenario, this group felt that each scenario must provide a constant output metric in order to evaluate that scenario or concept across maturity levels. The components identified by this breakout group were included in the scenario guidelines in Section 6. The eight other components from the original list were excluded from the group’s core list for a variety of reasons. Some components were too variable or specific to be considered core. For example, ‘Environment’ refers to weather (storms, lightning, etc.), noise, and emissions, which may or may not be present, as opposed to operational conditions such as instrument flight rules (IFR) and visual flight rules (VFR). Likewise, ‘Traffic Flow Management’, although becoming a more important role in the air traffic system, may or may not be present. ‘Other (Safety Situations, Security Situations)’ addresses specific needs that are not considered a core component. Some components are either not integrated in many fast-time models, or are at an inadequate level of maturity to be considered core components. For example, ‘Scripted Events’, ‘Technology Configuration’ (Communication/Navigation/Surveillance), and ‘Human Factors’ are more relevant in real-time simulations. Although ‘Aircraft Performance’ is a necessary component, the breakout group concluded that aircraft performance is part of a model, not part of a scenario, and therefore not considered a core component. This is especially true in fast-time models that do not allow changes to the aircraft performance calculations, which are embedded within the models. However, with the advent of fast-time models with open architectures, aircraft performance will likely develop into a core component. Finally, many of the ‘Assumptions and Constraints’ were either included in the operational procedures or are part of the experimental design. 4.4.2

Breakout Group 2 - “Typical” Real-Time Simulation Components

The goal of Breakout Group 2 was to identify ‘typical’ real-time simulation components in milestones V1-V3 of the Maturity Model. Like Breakout Group 1, they agreed that the maturity level corresponded to the fidelity of the simulation, where fast-time simulation is typically considered during milestones V1 or V2, and real-time in V2 or V3. Therefore certain components of a scenario can be shared across maturity levels, while the level of detail will increase. Before identifying the scenario components, the group recognized certain aspects of a realtime simulation that should be addressed. These include: •

Provide contact information for all involved in the study, including researchers, participants, and sponsors.

C-16

• • • • • • •

Define the objective of the study. Define the operational concept. Define the roles and responsibilities of the participants. Define the number of participants required. Determine whether specific or generic airspace will be used. Identify any special considerations. Define the assumptions and constraints.

After addressing these aspects, the researcher will have a clearer vision of the study and can better prepare the scenario components, as defined by Breakout Group 2. The group included seven core components on their list (Table 1). Core components for real-time simulation: • • • • • • •

Traffic Airspace Characteristics Airport Characteristics Operational Procedures Weather Scripted Events Technology/Equipment

C-17

Table 1. Core Components Level or Count - relative to normal operations or forecast as a percentage or multiple Traffic

Aircraft Fleet Mix - percentage of jets, props, general aviation, military, and special traffic Aircraft requiring different services or clearances - percentage of overflights, departures and arrivals Scenario Length En Route, Terminal, Tower, or Oceanic High or Low Sectors (#) Transitioning Aircraft

Airspace / Airport Characteristics

Military Operations Area (MOA)/Special Use Airspace (SUA) Radar Coverage Airspace Type - Managed vs. Unmanaged Airport Category Geographical Scope Runway Layout Specific Procedures Description

Operational Procedures

Phraseology Traffic Flow Management (TFM) Restrictions IFR/VFR Flights

Weather

Storms, Winds, Shear, Visibility

Scripted Events

Intentional Errors or System Failures Specific Description Communication/Navigation/Surveillance

Technology and Equipment

Decision Support Tools - Ground and Airborne Baseline Capabilities

This component list is similar to the Breakout Group 1 fast-time simulation list, however depending on the model or simulator; the component information may or may not be exchanged. One possibility for fast- and real-time information exchange is to take a subset from the fast-time simulation and apply it to the real-time simulation or vice-versa. For example, a real-time simulation that only requires a certain sector could be extracted from a larger fast-time simulation that may simulate the entire ECAC or NAS.

C-18

4.5

Sub-question 3 - ATM Objective Scenarios

Sub-question 3 was originally proposed, “Are ATM objective-specific scenarios needed and how should these be linked?” The workshop participants’ saw the need to address the issue from a slightly different perspective using the following related question as a guide; “Could the same scenario be used to meet several or all ATM objectives?” ATM objectives refer to the ATM performance objectives found in the OCVSD, which at the highest level include capacity, economics, safety, environment, national security, operability, uniformity, and training. Discussions among the practitioners centered on the idea that most scenarios developed for concept validation activities (regardless of the validation method used) are already ATM objective-specific or are used to meet more than one objective. They are however, uniquely developed for each validation activity (i.e., not “standard”). The current thinking is that developing standard ATM objective-specific scenarios may not be practical or useful at this time because the futuristic aspects of most concepts ultimately result in unique scenario requirements. In most cases, the combination of the operational concept, the validation method, the domain, and the ATM objectives drive requirements for scenario development. Decisions on whether objectives are met are generally based on assessment of the metrics gathered. Therefore, the practitioners suggested that it would be more feasible to consider a standard set of metrics for each ATM objective and ensure they are captured in the scenarios, rather than try to develop standardized scenarios for each ATM objective. Then, for example, if you are interested in examining capacity, you would design your scenarios to capture all the necessary/required standard capacity metrics (regardless of the validation method being used, the concept, the domain, etc). Several ATM objectives could be explored in the same scenario (e.g. capacity, safety, operability) by simply ensuring all corresponding standard metrics are captured (many, in fact, may overlap). In summary, developing standard ATM objective-specific scenarios is not recommended, but defining specific metrics for ATM objectives would facilitate a common way to develop scenarios and validate concepts. It should be noted that several organizations have been working on standard sets of metrics, including but not limited to; EUROCONTROL (via their INTEGRA project), NASA Ames (via their VAMS project), and the FAA.

4.6

Sub-question 4 - Validation Technique Scenarios

Sub-question 4, “Are validation technique-specific scenarios needed (e.g., fast-time or realtime simulation) and how should these be linked?” is based on the perceived inability to share scenario component information and results across various validation techniques. In order to share scenario components across validation techniques, at the very least, scenarios must have a common high level definition, see Figure 2. As scenario development matures through the validation process, the level of detail in the scenarios increases and often produces technique-specific scenarios. However, if executed properly, this progression provides common scenario components and promotes sharing and reuse across validation techniques. Since some details of the components may be represented differently, any variations must be clearly captured in documentation. In the context of the Maturity Model, validation technique-specific scenarios are needed when the chosen validation technique has been applied and the maturity level reaches milestones V2 and V3. Although most of the scenario components will be similar across various techniques, the scenarios may require some components specific to the technique. For example, many real-time human-in-the-loop simulations deal with phraseology, while C-19

fast-time simulations do not. Also, components are often customized for a specific technique, model, or simulation capability. For instance, airport layouts may be developed using measures of latitude/longitude for one model or technique, but another requires x/y coordinates. This may be problematic but does not mean that the information should not or can not be shared across techniques. Sharing component information across techniques and even across specific models may require some type of translation, especially when you approach the level of detail required for a specific model. Such translations must be documented and include any assumptions. Sharing scenarios across validation activities is also possible, though components may need to be modified or adjusted according to the concept. For example, if a scenario created to explore the affects of new large aircraft in the system were reused to study the affects of aircraft separation, the components defining the procedures would change. The issue of sharing results across validation techniques was not fully addressed at the workshop and the ability to share results in general was discussed at a high level. Therefore, further research is required for both these issues.

4.7

Sub-question 5 - Feasibility of Standard Scenarios

Breakout Group 3 drafted the expectations on the use of ‘standardised scenarios’ using a slightly modified version of question 5, “In what conditions would the use of standard scenarios be feasible or appropriate?” The group did not discuss the distinction between the uses of fast- and real-time simulations, therefore, further investigation concerning this topic is recommended. The group’s objective was to limit the discussion and development of standardised scenarios to one well described application that the all members of the breakout group could identify with. In addition, the application needed to address the following requirement as expressed by the FAA: “It is necessary to be able to compare the capabilities of any proposed concept in order to help decide where R&D investment could be best focused because development funding and expertise is limited and should be allocated according to some objective criteria.” The group addressed this need by defining input and output requirements for a ‘standardised baseline scenario” that any proposed concept could be evaluated against. The group agreed to define a ‘standardised baseline scenario’ as: “A predicted ‘future’ comprised of assumed demand, assumed supply capability, and assumed ATM constraints, which acts as a reference against which the performance and behaviours of any concept can be evaluated.” Table 2 lists the type of detail required to define a standardised baseline scenario.

C-20

Table 2. Baseline Scenario Input and Output Requirements Demand

Current or Forecast for specified year Unconstrained demand or constrained demand, e.g. expected availability of airports, use of secondary airports Range – highest to lowest forecast demand

Supply

ECAC/NAS wide application Technical performance Communications/Navigation/Surveillance

–

Infrastructure description Capacity capability ‘Concept in use’ description ATM Constraints

Environmental Human Resources

Airspace

Type Characteristics

Baseline Scenario Outputs Performance

Key Indicators and Metrics to be determined Capacity, Efficiency, Safety, Financial, Environmental

Behavioural

Operability and usability indicators and metrics for system operators (controllers, pilots, Airline Operations Centers (AOCs), etc.) Usability indicators and metrics for system user (airlines, military, GA, etc.) e.g. Ease of access to ATM system

Technical

Description of technical enablers and operational requirements on those enablers

Once developed, a standardized baseline scenario can significantly contribute to the validation process and provide valuable evidence required to support a specific concept. However, it is important to recognize that this is only one part of the validation process. When developing a baseline scenario, rather than assume current capabilities as a set point for the scenario, the practitioners agreed that assumptions should be made regarding a minimum capability level for the specified time in the future. The baseline scenario would therefore be ‘a stake in the sand’, which would not necessarily represent a valid future capability, but would be a benchmark against which to evaluate any concept. It would be the responsibility of an organisation’s management to define the detail of the baseline scenario for that projected timeframe.

C-21

One approach would be to develop a baseline scenario for a specific timeframe based on traffic forecasts. For example, the EUROCONTROL Air Traffic Statistics and Forecasts (STATFOR) Unit regularly publish medium-term and long-term flight forecasts based on the anticipated growth of traffic between city pairs for the entire ECAC area (available at www.eurocontrol.int/statfor). These flight forecasts can be translated into traffic patterns for specific airspace (e.g., sectors). A tool such as the Traffic Sample Generator (TSG) from EUROCONTROL’s CARE INTEGRA project could then be used to build and populate the scenario. The TSG maps the traffic on the airspace and "cuts out" the relevant traffic for a selected set of sectors in a format suitable for fast- or real-time simulators. See for further http://www.eurocontrol.int/integra/public/standard_page/INTEGRA_tsg.html information.

5. CARE/ASAS CASE STUDY The purpose of this section was to provide an example of the use of the CARE/ASAS scenario template to explore the basic requirements for defining a scenario. The CARE/ASAS action was to study under which circumstances the use of Airborne Separation Assistance Systems would allow the transfer of responsibility of separation assurance from the Air Traffic Controller to the pilot (see http://www.eurocontrol.int/care/public/subsite_homepage/homepage.html). Activities have been performed covering aspects like assessing the dimension of the problems to be encountered, defining an ASAS validation framework, defining factors influencing airborne separation minima and studying delegation issues and safety analysis of a number of ASAS applications envisaged to come in operation in the future. CARE/ASAS developed templates for scenario development in fast-time simulations that are available at http://www.eurocontrol.int/care-asas/gallery/content/public/docs/act2/care-asasa2-02-030.pdf The CARE/ASAS templates provide a checklist of scenario components for a given application. These templates have been broadly discussed in CARE/ASAS related workshops and have been generally well received by the participants. To provide some insight into these templates, the following case study was conducted. The CARE/ASAS template for validation of ASAS Applications was useful for defining the ASAS/S2 Enhance Sequencing and Merging (S&M) Operations Application. It provided information on the new tasks that the S&M application implies, as well as giving the type of information needed to prepare any simulation model. However, more detailed information is needed to define the fast-time simulation scenarios, for example, characteristics of traffic to be used, geographical area in which S&M is to be applied, etc. The following information identifies the usefulness of each aspect of the template for the definition of the fast-time simulation scenario.

ASAS Application and Associated Flight Phases: This aspect defines the application of the concept during each phase of flight. Most of the applications were listed; however some were missing (e.g. S&M).

High-Level Objectives: Identifying the high level objectives are useful, however the definition of more detailed metrics would have been helpful.

Airspace: All the components required for airspace modeling were identified; however they needed to be defined in more detail.

C-22

Traffic: All the components required for developing traffic were identified; however they needed to be defined in more detail. It was assumed that the field “trajectory” includes definition of full path, e.g.: Origin/Destination Airport, Flight Levels requested and Flight Path.

Air Traffic Services Involved: The high level definition of the actors involved was useful, however more detailed information of the actors involved would be useful.

Rules: All the rules required for this application were identified; however they needed to be defined in more detail.

Tasks/Actors: New tasks were defined but more detailed information is needed to define the operational concept

Technology: Definition of the operational concept depends on the technology used. Further information on how the new technology works, from an operative point of view, is needed.

Tasks/Actors/Technology: More detailed information was needed to define the operational concept, the operative functionalities of new technologies, tasks, and actors.

This case study was not extensive; however, it suggests that the CARE/ASAS template could be useful for the scenario definition if more components are supported in the template.

6. SCENARIO GUIDELINES Though not a formal part of the workshop, the scenario guidelines presented as an attachment to this appendix were developed based on input from the workshop and other sources of information including CARE/ASAS. The goal is to provide practitioners with a tool/template to ensure scenario completeness during scenario development. It is envisioned that the guidelines could be used as an extension of, or supplement to, the experimental plan of a validation exercise. A completed guideline could also provide the sponsor with a log of the scenario components and assumptions for the validation exercise, which can be used as a guide for further research on the same concept. There may be instances in which these guidelines are not applicable. For example, the guidelines do not attempt to quantify every component found in a scenario. Certain components may not be included in this list as they may pertain to an emerging technology or concept that has not yet been fully developed. Likewise, depending on the technique used for the validation, certain components may not be relevant; for example, phraseology does not apply to fast-time simulation. However, one should consider populating any component if the data is available. This would allow for data exchange and reuse between other techniques. One guideline should be used for each demand or exercise, thereby allowing specific information for each scenario to be documented. For example, practitioners should complete a guideline for a baseline case to show what technologies, rules etc., were used; then they should complete another template for a future case or alternative case to document the new technologies or traffic levels used. A final version of the guidelines will become a downloadable electronic tool that practitioners can utilize to indicate components used during each scenario. The details within certain components will change based on the addition of new technologies, the location (U.S. or

C-23

Europe), and the implementation of new concepts. Full instructions will be provided with the final scenario guidelines. An outline and short description of the guidelines is provided as an attachment to this appendix.

7.

CONCLUSIONS AND RECOMMENDATIONS

The use of standard scenarios is an attempt to provide comparability among validation results. Although comparability of results is significant to external stakeholders who need to compare concepts at an early stage of development to make investment decisions, validation practitioners prefer customized exercises that allow specific performance issues to be measured. It is this gap, between external stakeholders and practitioners, which impeded efforts to address the workshop objective and limited discussions of the subquestions. To the extent possible, the authors’ expertise was used to present a thorough discussion of the sub-questions. Workshop attendees accepted the use of the Action Plan 5 Concept Maturity Model as a framework for developing a typical, though not standard, scenario. They concluded for both validation techniques (real-time and fast-time) that at each level of maturity scenario components remain constant, but the information contained therein may vary. At a minimum each scenario should contain traffic data, airport, and airspace characteristics and the applicable operational procedures. Additional investigation of the Concept Maturity Model is recommended to determine the type of experiment and assessment should be applied during the stages of development. The workshop attendees also recommend the application of a standard set of metrics for specific ATM objectives. Since advances in standardizing scenarios have been limited, but advances in standard indicators have not (e.g., INTEGRA), the definition of metrics for specific ATM objectives could possibly facilitate the development of standard scenarios. Lastly, the workshop discussions recommend the use of a standardized baseline scenario following the direction of the corresponding scenario guidelines. The standardized baseline scenario should be used as a benchmark to evaluate any concept rather than represent a valid future capability. Using the scenario guidelines will ensure scenario completeness during development. In summary, the workshop was successful in identifying where further investigation is needed for developing standard scenarios. Additional research should be conducted for the refinement of the Maturity Model and the establishment of standard metrics before a serious discussion of standard scenarios can occur. Collectively this research can serve as the basis for establishing standard scenarios and a process for developing scenarios throughout the Maturity Model lifecycle.

C-24

Abbreviations AENA

Aeropuertos Españoles y Navegación Aérea

AMASS

Airport Movement Area Safety System

AOC

Airline Operations Center

AP5

Action Plan 5 - Validation and Verification Strategy

AP9

Action Plan 9 - Air Traffic Modeling of Operational Concepts

ARTS

Automated Radar Terminal System

ASAS

Airborne Separation Assistance

ATM

Air Traffic Management

CAMI

Civil Aerospace Medical Institute

CARE

Co-operative Actions of R&D in EUROCONTROL

CDIMS

Collaborative Discussion and Information Management Service

D2

Direct-to

ECAC

European Civil Aviation Conference

E-OCVM

European - Operational Concept Validation Methodology

FAA

Federal Aviation Administration

FMS

Flight Management System

GA

General Aviation

HCS

Host Computer System

HMI

Human-Machine-Interface

IFR

Instrument Flight Rules

JPDO

Joint Planning and Development Office

LDA

Localizer-Type Directional Aid

C-25

MOA

Military Operations Area

NAS

National Airspace System

NASA

National Aeronautics and Space Administration

NDB

Non-Directional Beacon

NLR

Nationaal Lucht en Ruimtevaartlaboratorium

OCVSD

Operational Concepts Validation Strategy Document

OEP

Operational Evolution Plan

R&D

Research & Development

RNP

Required Navigational Performance

RVSM

Reduced Vertical Separation Minima

S&M

Sequencing and Merging

SME

Subject Matter Expert

STATFOR

Statistics and Forecasts

SUA

Special Use Airspace

TAF

Terminal Area Forecasts

TFM

Traffic Flow Management

TSG

Traffic Sample Generator

UAS

Unmanned Aircraft Systems

VDR

Validation Data Repository

VFR

Visual Flight Rules

VOR

Very High Frequency Omnidirectional Range

WARP

Weather and Radar Processor

C-26

References Validation Data Repository: User Guide. EUROCONTROL, 2004. Edition 2.0 https://www.eurocontrol.int/eatmp/vdr/doc/VDR_quickstart.pdf FAA/EUROCONTROL COOPERATIVE R&D: Action Plan 5 Operational Concept Validation Strategy Document. December 8, 2003. Edition Number 1.3, http://www.eurocontrol.int/valug/gallery/content/public/OCVSD%20-V1-3.pdf CARE/ASAS Activity 2: VF Project-WP1-D1. Initial Validation Framework and Scenario Report Version 1.0 - 08 August 2002 http://www.eurocontrol.int/care-asas/gallery/content/public/docs/act2/care-asas-a202-031.pdf Makins N., EUROPEAN Operational Concept Validation Methodology, Eurocontrol, December 14, 2004 V1.0 –https://www.eurocontrol.int/eatmp/vdr/doc/OCVM.pdf

C-27

Attachment C1 - Scenario Guidelines These guidelines were developed from information contained in the CARE/ASAS templates and data gathered during the breakout groups of the workshop. The details within certain components will change based on the addition of new technologies, user location (U.S. or Europe), and the implementation of new concepts. The final version of these guidelines will be formatted in an electronic version and will supply full instructions for use. Practitioners’ will be able to fill in information using a text box or check boxes. A. Objective of the Scenario The objective of the scenario describes the intended purpose of the scenario. B. Application (specific to the program/project) & Associated Flight Phases a. In the Air i. Climbing ii. Cruise iii. Descending b. On the Ground The application section defines how the specific program or project is applied to various phases of flight. The section will provide insight into how a new concept or the baseline case operates. For example, if the project is to validate a new taxiway lighting tool, an application of this project could be; improved taxiway navigation in low visibility conditions on the ground. C. Traffic a. Schedule - Flight Plan i. Entry Time ii. Aircraft ID iii. Aircraft Model iv. Aircraft Equipment v. Trajectory and/or Flight Plans vi. Origin/Destination Pairs b. Level/Volume i. Very High ii. High iii. Medium iv. Low c. Aircraft Fleet Mix (%)* (ex. Heavy, B757, Large, Small) d. Aircraft Classification i. Airline ii. General Aviation (GA) iii. Military iv. Special v. Unmanned Aircraft Systems (UAS) e. Aircraft Equipage i. Equipped ii. Unequipped iii. Mixed iv. Percent equipped f. Aircraft Requiring Different Services/Clearances i. Overflights

C-28

ii. Departure/Arrival g. Scenario Length h. Time Frame/Forecast (%) i. Baseline ii. Future 1 (Define) iii. Future 2 (Define) iv. Other The traffic schedule at a minimum should include the time of entry, aircraft ID, aircraft model, and route. In this case; route is defined as a pre-defined trajectory or flight plan. Origin/destination pairs are only required when considering airports as part of the validation. However, if the data is available; gather it for possible reuse in other validations. In most validation activities, the baseline traffic schedule should be obtained during a high to very high traffic volume in order to stress the system significantly to evaluate the concept. Future schedules should correspond to specific target levels specified by the FAA or EUROCONTROL. For example, the FAA Flight Plan performance target is 2009, the FAA Operational Evolution Plan (OEP) performance target is 2013, and the Joint Planning and Development Office (JPDO) performance target level is 2025. If possible, three traffic levels should be chosen for a study in order to create a 3 point plot that shows increasing levels of capacity, delays, and so forth. In the U.S., target levels of traffic should coincide with the FAA Terminal Area Forecasts (TAF) for the given target year. Aircraft fleet mix and classifications can be obtained from the schedule. Aircraft equipage should be considered when evaluating operations were equipage changes operational procedures. In real-time simulation, it is important to be aware of aircraft requiring different services/clearances. This component can affect the setup of the simulation. For example, non-Reduced Vertical Separation Minima (RVSM) equipped aircraft operating in an RVSM environment. The scenario length depends on a number of factors, such as availability of controllers, tool limitations, scope, etc., and will not be discussed here in detail. Typically in fast-time simulation the scenario is run for a 24-hour period, while realtime simulation lasts from 15 minutes to 1 hour. D. Airspace Characteristics a. Type i. Site Specific ii. Generic b. Geographical Scope i. Global ii. United States (U.S.) (ex, NAS/CONUS) iii. Europe (Central, Eastern Europe, etc.) iv. Site Specific c. Airspace Classes i. Controlled (Classification) ii. Uncontrolled d. Areas

C-29

i. En Route ii. Terminal iii. Oceanic e. Sectors (#) i. En Route High ii. En Route Low iii. Terminal iv. Oceanic f. Military Operations Area (MOA)/Special use airspace (SUA) g. Route Structure i. Required Navigational Performance (RNP) ii. Waypoints iii. Alternative Routing iv. Descent/Ascent Profiles v. Playbook vi. Free Flight h. Flight Levels i. Terrain Topography j. Environmental i. Weather 1. Storms 2. Winds/Shear 3. Visibility ii. Atmospheric Conditions iii. Noise iv. Emissions Airspace characteristics determine the scope and elements required for a validation that involves the airspace. In most cases site specific information is obtained; however, as mentioned in the OCVSD document, generic airspace can be a valuable tool in concept validation. In general, controlled airspace is typically the class of airspace that will be examined. Airspace areas are usually selected based on the concept, however one must consider the implications on the different airspace areas that a specific concept may have. For example, a terminal area enhancement may cause a procedural change for the en route area. E. Airport Characteristics a. Airport Layout i. Runways ii. Taxiways iii. Gates iv. Exits b. Tower c. Terrain Topography d. Airport Category i. Cat I ii. Cat II iii. Cat IIIa iv. Cat IIIb e. Environmental i. Weather

C-30

1. Storms 2. Winds/Shear 3. Visibility ii. Atmospheric Conditions iii. Noise iv. Emissions Airport characteristics determine the scope and elements required for a validation that involves airports. F. Operational Procedures a. Air Traffic Services i. Air ii. Ground b. Specific Procedures Description c. Controller Phraseology i. Standard ii. New (define) d. Pilot Phraseology i. Standard ii. New (define) e. Traffic Flow Management (TFM) Restrictions i. Airspace ii. Airport f. Airspace Separation i. Time ii. Distance g. Airport Separation i. Time ii. Distance h. Aircraft Sequencing i. Flight Rules i. Instrument Flight Rules (IFR) ii. Visual Flight Rules (VFR) iii. New j. Reduced Vertical Separation Minima (RVSM) i. Yes ii. No Operational procedures include components such as air traffic services, phraseology, TFM restrictions, separations, sequencing, and flight rules. The operational procedures component should define the operational environment the concept will work in. G. Technology/Equipment a. Airspace Ground Technology/Equipment i. Communications 1. Voice/Frequencies 2. Data Link 3. Inter-sector information transfer 4. Other ii. Surveillance/Flight Data Processing (ex. ARSR, ADS-B)

C-31

b.

c.

d.

e.

iii. ATM 1. Planning 2. Automation 3. Host Computer System (HCS) 4. Other iv. Navigation (ex. Very High Frequency Omnidirectional Range (VOR), Non-directional Beacon (NDB), Distance Measuring Equipment (DME)) v. Display (ex. Display System Replacement (DSR), Standard Terminal Automation Replacement System (STARS)) vi. Decision Support Tools (ex. User Request Evaluation Tool (URET), Traffic Management Advisor (TMA), Direct-to (D2)) Airspace Airborne Technology/Equipment i. Communications 1. Voice/Frequencies 2. Data Link 3. Other ii. Surveillance/Flight Data Processing (ex. Mode-S, Traffic Alert and Collision Avoidance System/Resolution (TCAS)) iii. Navigation (ex. Global Positioning System (GPS), Flight Management System (FMS), RNP) iv. Display (ex. Calculated Delay Interval (CDTI)) v. Decision Support Tools Airport Ground Technology/Equipment i. Communications 1. Voice/Frequencies 2. Data Link 3. Other ii. Surveillance/Flight Data Processing (ex. ARTS, STARS, Airport Movement Area Safety System (AMASS)) iii. ATM 1. Planning 2. Automation 3. Other iv. Navigation (ex. Instrument Landing System (ILS), VOR, LocalizerType Directional Aid (LDA), Microwave Landing System (MLS)) v. Display vi. Decision Support Tools (ex. Surface Movement Advisor / Surface Management System) Airport Airborne Technology/Equipment i. Communications 1. Voice/Frequencies 2. Data Link 3. Other ii. Surveillance/Flight Data Processing (ex. Mode-S) iii. ATM iv. Navigation (ex. GPS) v. Display (ex. CDTI) vi. Decision Support Tools Weather Systems (ex. Weather and Radar Processor (WARP), Integrated Terminal Weather System (ITWS)

C-32

Technology/Equipment defines the technologies and equipment that will be used in the scenario. This includes both ground and airborne based technologies that support airspace and airport operations. This section provides a checklist of current and future technologies in the area of communications, surveillance, ATM, navigation, displays, and decision support tools. This section will be updated when future technologies are introduced into the system. H. Traffic Flow Management The TFM component should be used to describe any TFM issues not related to TFM restrictions, such as Collaborative Decision Making between the TFM and an Airline Operations Center (AOC). I.

Aircraft Performance

Aircraft performance is a necessary component of a scenario; however it is typically not altered or defined by the practitioner. In real-time simulation, aircraft performance is usually a high fidelity representation of an aircraft and is defined by the software that drives the simulation. In fast-time simulation, aircraft performance is typically embedded within the models. However, with the advent of fast-time models with open architectures, aircraft performance can be represented in high or low fidelity. J. Output Metrics Output metrics, although an important scenario component, will not be described in this document. Practitioners must assure that the metrics collected align with the high level objectives. If possible, collect more data than required for future analysis. This is easier in fast-time since most of the metric calculations are completed during post-processing and do not have the same data storage and processing speed issues that apply to real-time simulation. K. Actors a. b. c. d. e. f. g. h. i. j. k.

A-Side/Tracker R-Side D-Side Flight Crew Terminal Controllers Tower Controllers Supervisor Traffic Management Coordinator Airline Operations Center Technology Other

This scenario component provides a list of actors needed to validate the concept. Typically, this component is essential for real-time simulation and less so for fasttime. However, some fast-time models do consider the actors. Coordination with subject matter experts (SMEs) is required to determine the number of positions required during validation. L. Scripted Events (ex. Intentional Errors/System Failures) A scripted event deliberately introduces a disturbance into the system. In real-time simulation this disturbance is generally known in advance to the validation team but

C-33

not the simulation participants. In fast-time simulation the disturbance is generally introduced at a particular time. M. High Level Objectives a. Capacity i. Throughput ii. Access iii. Predictability iv. Flexibility v. Airport Capacity vi. Efficiency b. Economics i. Service Charges ii. Equipment Costs iii. Flight Efficiency iv. Operating Costs c. Safety i. Risk Identification ii. Risk Classification iii. Risk Management iv. Risk Monitoring d. Environment i. Gas Emissions ii. Noise e. National Security f. Operability i. Usability g. Uniformity i. Interface Standards h. Training i. Human Involvement & Commitment The high level objectives define the objective of the scenario. These objectives should be customized for the concept and coincide with the output metrics. N. Other (Safety Situations, Security Situations) This component is a general catch all to document specific situations related to the validation. O. Assumptions and Constraints Assumptions and constraints are not necessarily a scenario component; however they must be considered and listed during validation activities.

C-34



Appendix D ICAO’s Key Performance Areas

This is a list of ICAO’s published Key Performance Areas.

KPA 01 KPA 02 KPA 03 KPA 04 KPA 05 KPA 06 KPA 07 KPA 08 KPA 09 KPA 10 KPA 11

Access and Equity Capacity Cost Effectiveness Efficiency Environment Flexibility Global Interoperability Participation by the ATM Community Predictability Safety Security

D-1

Operational Concept Validation Strategy Document

Operational Concept Validation Strategy Document

Suggest Documents

GEO BON Concept Document

ICSC strategy document - DeSmogBlog

companies - DOCUMENT Strategy Media

PROPOSED STRATEGY DOCUMENT

companies - DOCUMENT Strategy Media

Concept-based document indexing

concept document - Filene Research Institute

concept document - Filene Research Institute

Project Background Problem Statement Final Concept Operational ...

ICAD5092B Update and document operational procedures

Operational Requirements Document (ORD-Update) - DLR

XML Document Parsing: Operational and ...

DEVELOPMENT AND VALIDATION OF AN OPERATIONAL ...

Validation of MIPAS-ENVISAT H2O operational data

Strategy Document 2016 - Composites UK

Strategy discussion document - Spinal Research

strategy document - Thrive Nova Scotia

strategy document - Thrive Nova Scotia

(CERES) Validation Document Validation of Imager Cloud Optical ...

(CERES) Validation Document Validation of Imager Cloud Optical ...

Overall operational Strategy - Action contre La Faim

SGP 6th Operational Phase Country Programme Strategy ...

Overall operational Strategy - Action contre La Faim

New Document Concept and Metadata Classification ...