NATO UNCLASSIFIED SCIENCE AND TECHNOLOGY ORGANIZATION CENTRE FOR MARITIME RESEARCH AND EXPERIMENTATION
Technical Report
CMRE-FR-2012-009
Knowledge elicitation for fuzzy rulebased decision support systems and system interface evaluation and design
Yvonne R. Masakowski, Raffaele Grasso December 2012
This document is for distribution only to NATO, Government Agencies of NATO member nations and their contractors. Requests for secondary distribution shall be made to the Science and Technology Organization - Centre for Maritime Research and Experimentation (STO-CMRE).
Official Information
NATO UNCLASSIFIED
No Public Release
About CMRE
The Centre for Maritime Research and Experimentation (CMRE) is a world-class NATO scientific research and experimentation facility located in La Spezia, Italy. The CMRE was established by the North Atlantic Council on 1 July 2012 as part of the NATO Science & Technology Organization. The CMRE and its predecessors have served NATO for over 50 years as the SACLANT Anti-Submarine Warfare Centre, SACLANT Undersea Research Centre, NATO Undersea Research Centre (NURC) and now as part of the Science & Technology Organization. CMRE conducts state-of-the-art scientific research and experimentation ranging from concept development to prototype demonstration in an operational environment and has produced leaders in ocean science, modelling and simulation, acoustics and other disciplines, as well as producing critical results and understanding that have been built into the operational concepts of NATO and the nations. CMRE conducts hands-on scientific and engineering research for the direct benefit of its NATO Customers. It operates two research vessels that enable science and technology solutions to be explored and exploited at sea. The largest of these vessels, the NRV Alliance, is a global class vessel that is acoustically extremely quiet. CMRE is a leading example of enabling nations to work more effectively and efficiently together by prioritizing national needs, focusing on research and technology challenges, both in and out of the maritime environment, through the collective Power of its world-class scientists, engineers, and specialized laboratories in collaboration with the many partners in and out of the scientific domain.
Copyright © STO-CMRE 2012. NATO member nations have unlimited rights to use, modify, reproduce, release, perform, display or disclose these materials, and to authorize others to do so for government purposes. Any reproductions marked with this legend must also reproduce these markings. All other rights and uses except those permitted by copyright law are reserved by the copyright owner. Single copies of this publication or of a part of it may be made for individual use only. The approval of the CMRE Information Services is required for more than one copy to be made or an extract included in another publication. Requests to do so should be sent to the address on the document data sheet at the end of the document.
CMRE-FR-2012-009
NATO UNCLASSIFIED
Knowledge elicitation for fuzzy rulebased decision support systems and system interface evaluation and design
Yvonne R. Masakowski, Raffaele Grasso This document, which describes work performed under the Environmental Knowledge and Operational Effectiveness programme of the CMRE Scientific Programme of Work, has been approved by the Director. This work relates to Department of the Navy Grant N62909-12-1-7083 issued by Office of Naval Research Global. The United States Government has a royalty-free license throughout the world in all copyrightable material contained herein..
-i-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Intentionally blank page
- ii -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Knowledge elicitation for fuzzy rule based decision support systems and system interface evaluation and design
Yvonne R. Masakowski, Raffaele Grasso
Executive Summary: This report summarizes research efforts conducted as part of the US Navy International Cooperative Opportunities Program (NICOP) research awarded by ONR Global to NUWCDIVNPT and CMRE, for the development of CMRE’s Goal Oriented-Decision Support System (GO-DSS) framework. The aim of this work was to evaluate CMRE’s METOC Tactical Decision Aid (TDA) system which is part of the GO-DSS framework and make recommendations for enhancing the design and capabilities of the visual spatiotemporal interface display. The NICOP research effort is part of a larger NATO (CMRE) research program known as the “Environmental Knowledge and Operational Effectiveness (EKOE)” program. This specific research effort is in support of the EKOE-3 project, also known as “Decisions in Uncertain Ocean Environments”. The NICOP research topic area focuses on the design and development of a tactical decision aid which is part of the GO-DSS framework. The GO-DSS framework includes tools to assess the impact of uncertainty on decision making as related to environmental information (i.e., meteorological and oceanographic) and nonenvironmental information (i.e., traffic density) involved in mission planning and to deploy assets in the maritime operational environment. Environmental variables considered in the GO-DSS framework include wind velocity, current speed and wave height, which are forecasted by a deterministic and/or a probabilistic forecast system (like a Super Ensemble (SE), which fuses the forecasts provided by multiple forecasting centers). The GO-DSS framework is currently under development and is a concept for a future GEOMETOC information management, decision support, and visualization capability. Existing systems currently lack comprehensive tracking of uncertainties to ensure proper consideration of risks in the decision making process. Moreover, competing, conflicting mission objectives, models and potentially contradictory information can prevent timely and objective decision making, and thereby increase the risk of operational effectiveness. The objective is to develop an enhanced integrated GO-DSS framework to increase the automation, and effectiveness of information fusion, environmental impact and uncertainty assessment, and decision making processes. The result will be an enhanced decision support tool for METOC operations intended to support US Navy and NATO maritime operations.
- iii -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Intentionally blank page
- iv -
NATO UNCLASSIFIED
CMRE-FR-2012-009
Knowledge elicitation for fuzzy rule based decision support systems and system interface evaluation and design
Yvonne R. Masakowski, Raffaele Grasso
Abstract: This paper summarizes the process for the design and development of a fuzzy rule-based tactical decision aid (FR-TDA) system which is part of the CMRE’s GO-DSS framework and the design of an enhanced interface to support decision making and maritime operation planning. The FR-TDA system under study will provide the user with geospatial meteorological and oceanographic (METOC) impact surfaces as well as risk and decision confidence indexes to support a wide spectrum of maritime operations. In this work, underwater glider deployment and recovery operations are considered as a case study. There are a number of issues involved in selecting the optimum path for glider deployment and recovery. Among these issues, it is critical to attain an understanding of the meteorological and oceanographic environment in which the gliders are operating. This study is aimed at developing a decision support tool that will assist the METOC operator with planning an optimal course of action in environmental operations by means of evaluating the environmental picture to optimize glider performance. There are two significant Human Factors (HF) issues related to developing such a decision support system (DSS): 1) de-conflicting uncertainty and 2) reduce time to decision making while increasing accuracy. In order to address the first issue, it is essential to identify the critical factors/cues used by experienced METOC Subject Matter Experts (SMEs) during their evaluation of METOC environmental data embedded with noise, clutter and uncertainty. Knowledge elicitation of SMEs for system rule bases from domain experts will facilitate mapping information usage of METOC environmental information into the DSS. The design of the system interface will require information from SMEs regarding the level of uncertainty and operational constraints and the level of significance of individual variables which they integrate during their decision making processes. The design of geographic information system (GIS) interfaces and assessment of their effectiveness is aimed at providing both an increase in system and human performance. That is, the ultimate goal is to design a DSS that optimizes the path for the glider, as well as optimizes the human’s decision making processes. This study will explore the METOC expert’s approach to mission planning for glider operations and their ability to assess and predict the environmental impact on maritime operations and provide an optimal course of action by means of METOC forecast models, as well as facilitate the development of an intuitive interface, aimed at integrating time as a critical variable for decision making.
Keywords: Decision support, fuzzy logic, uncertainty, and interface design.
-v-
CMRE-FR-2012-009
NATO UNCLASSIFIED
Contents 1.
Introduction ............................................................................................................ 1 1.1 Purpose and Scope ........................................................................................... 1
2.
Fuzzy logic tactical decision aid ............................................................................ 4 2.1 System overview .............................................................................................. 4 2.2 The Decision Support Tool .............................................................................. 5
3.
NICOP: Tasks 1 and 2 ......................................................................................... 12 3.1 Background: Scope, Limitations and Assumptions ....................................... 12 3.2 Introduction.................................................................................................... 13 3.3 Research Objectives: Review of the CMRE FR-TDA system ...................... 13 3.4 Technical Approach: Evaluation of the Interface Design ............................. 14 3.5 Visual Analytics and Implications for Interface Design ................................ 18 3.6 Conclusions & Recommendations ................................................................. 19
4.
NICOP : Task 3 .................................................................................................... 21 4.1 Background: Scope, Limitations and Assumptions ....................................... 21 4.2 Introduction.................................................................................................... 22 4.3 Technical Approach ....................................................................................... 22
5.
NICOP: Task 4 ..................................................................................................... 25 5.1 Technical Approach: ...................................................................................... 25 5.2 Conclusions and Recommendations .............................................................. 26
6.
NICOP: Task 5. .................................................................................................... 27 6.1 Conclusions and Recommendations .............................................................. 27
Acknowledgement ............................................................................................................ 31 References ......................................................................................................................... 32 Annex A: NICOP amendments ......................................................................................... 35 Annex B: FR-TDA questionnaire 1 .................................................................................. 36 Annex C: FR-TDA questionnaire 2 .................................................................................. 37
- vi -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
1
Introduction The NICOP final report describes the scope of the proposed research, the technical approach and range of issues addressed in support of the further development of the CMRE mission planning framework known as Goal-Oriented Decision Support System (GO-DSS). The NICOP proposal was organized in 5 tasks as follows: T1. Review of the CMRE fuzzy rule based TDA system application and products (NUWC, CMRE). T2. Specify future design lines to follow in order to improve the interface and the products presentation (NUWC). T3. Outline a protocol for knowledge elicitation that is specific for fuzzy rule based systems (NUWC, CMRE). T4. Outline a system evaluation protocol that involves the participation of potential system users (NUWC). T5. Specification of an operational domain and test protocols with a pool of subject experts (NUWC, CMRE). T5 was seen as a medium/high risk activity as it depended on the availability of a relevant pool of experts. The recommendation was made to complete Tasks 1-4 as these tasks did not involve human subject research. In addition, authors were directed to draft a research protocol for use in upcoming CMRE experiments to evaluate the GO-DSS effectiveness for METOC operators conducting mission planning under conditions of uncertainty. Appendix A reports details on NICOP amendments.
1.1 Purpose and Scope The purpose of the CMRE/NUWC NICOP is to apply Human Factors methodology to the development of CMRE’s GO-DSS. The objective result is to design an intuitive, taskcentred interface for the GO-DSS system that will reduce operational risk and optimize human and operational performance. CMRE is leading a research program known as the “Environmental Knowledge and Operational Effectiveness” (EKOE) program, which is designed to support implementation of the rapid environmental assessment (REA) and recognised environmental picture (REP) concepts. Specifically, Project 3 under the EKOE program, -1-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
“Decisions in Uncertain Ocean Environment” (DUOE-EKOE3) focuses on conducting applied research in the field of decision support for operational planning in the presence of environmental uncertainty Its main goal is the design and testing in specific application domains of a GO-DSS architecture framework that is a concept for the future GEOMETOC information management, decision support and visualization capability. Existing systems currently lack comprehensive tracking of uncertainties to ensure proper consideration of risks in the decision making process. Moreover, competing, conflicting mission objectives and potentially contradictory information can prevent timely and objective decision making, and thereby increase the risk of suboptimal operational effectiveness. The GO-DSS is a multi-objective optimization approach which allows the integration of potentially conflicting cost functions and risk assessments that provide the decision maker with the ability to explore the full spectrum of solutions in the problem space. The GO-DSS framework contains tools to assess the impact of both environmental conditions (i.e., meteorological and oceanographic) and non-environmental conditions (e.g., traffic density maps) on developing a course of action and mission planning that will ensure safety of both people and assets. There are a variety of problems related to decision making under conditions of uncertainty, including selecting the optimal course of action in dynamic environmental operations using meteorological and oceanographic (METOC) data. The GO-DSS framework includes a fuzzy rule-based DSS to evaluate environmental uncertainty and integrate risks associated with a selection of variables (e.g., wind velocity, wave height, current speed, etc.) to facilitate the development of an operational plan that will reduce risk and increase safety in maritime operations. One of the main advantages of fuzzy rule-based systems is the flexibility in dealing with subjective and objective knowledge and vague concepts [1]. A fuzzy logic system can be trained using input-output examples which may be the result of a repeated experiment or by embedding subjective knowledge elicited from an expert or specific domain experts, such as METOC SMEs. Moreover, the combination of fuzzy rule-based Bayesian methods and other non-linear probabilistic techniques can be used in a hybrid manner to fuse knowledge of different kinds. The objective of this study is to develop recommendations and requirements for the enhanced design of the FR-TDA system in the GO-DSS framework and in particular to increase the automation and effectiveness of information fusion, environmental impact and uncertainty assessment, and decision making processes. The result will be an enhanced decision support tool for METOC operations intended to support US Navy and NATO maritime operations including, for instance, glider operations, ASW, MCM and counter piracy. The overarching goals of this project include an analysis and assessment of the FR-TDA system interface, outline of a procedure for gathering information from METOC SMEs and recommendations to improve the design of the system interface. Environmental variables such as wind velocity, current speed and wave height predicted by a probabilistic forecast system, like a Super Ensemble (SE) model, are taken as input into the FR-TDA. The uncertainties associated with the DSS inputs (related to differences among/between models in the ensemble) are propagated through the DSS output by using specific statistical tools such as the unscented transform [2]. In this way the system is
-2-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
able to provide a traffic light map (run vs. not run the operation) and also specifies a confidence level associated with each action. Evidence of the predictive success of the FR-TDA system may be found in the results of the CMRE Recognised Environmental Picture 2010 (REP10) experiment [1]. The REP 10 experiment demonstrated the capabilities of the system coupled with a glider track prediction model to provide surfacing and vehicle recovery options along the predicted path with low environmental impact. Although the original study successfully demonstrated the effectiveness for developing optimized pathways for glider operations, the experiment did not evaluate the METOC operator’s interactions with the system display interface, or provide insight as to the strategies used by METOC officers for evaluating risks and preferences. Achieving an understanding of the METOC operator’s interactions and strategies employed while using the FR-TDA system is critical to the development of an enhanced, intelligent GEOMETOC DSS. The current study focuses on the ways in which the METOC operator uses the FR-TDA system, how the interface supports their mission planning and decision making, and provide insights essential for the development of design requirements for future GEO-METOC DSSs. Thus, the present study was conducted to evaluate the METOC operator and the effectiveness of the FR-TDA current system display. The aim was to gain an understanding of the variables contributing to the development of recommended courses of action and to determine requirements for the development of future intuitive interface designs that would support decision making more effectively in the maritime environment. Lastly, this investigation adopted a cognitive engineering approach to evaluate human performance, assess METOC operator’s mental models of decision making and recommend the enhancement of the current FR-TDA system interface display. The report is organised as follows. Chapter 2 introduces the FR-TDA architecture giving a brief overview of the fuzzy logic inference engine used in the system. Chapter 3 describes the work done for task 1 and 2 of the NICOP while task 3 and 4 are reported in chapters 4 and 5 respectively. Conclusions and future work are outlined in chapter 6.
-3-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
2
Fuzzy logic tactical decision aid 2.1 System overview The FR-TDA system implemented at CMRE for supporting maritime operations is an extension of a previous architecture introduced in [3]. The goal of the proposed system is to help the decision authority (DA) in making decisions about actions related to the current operation at sea. The FR-TDA system assumes that a set of forecasts is available, related to the set of environmental variables which affect the maritime operation under consideration, for the area of interest before the planned operation takes place (typically the forecasting centres provide three days ahead forecasts with a time resolution of one to three hours). In addition the FR-TDA system requires a set of pre-defined actions related to the maritime operation to be provided beforehand (e.g.,, run mission, reschedule mission). The Decision Support Tool (DST), the core of the FR-TDA system (see Figure 1), handles the environmental forecasts and, based on a Bayesian risk assessment framework, computes the recommended action and the related risk. Once the recommended action and the related risk have been computed for each point falling inside the area of interest, they are injected into a GIS server. A GIS client is used to provide maps with recommended actions and related risks to the DA, who can navigate them in space and time. In the next section we describe the DST in detail. METOC centre
environmental forecasts
Decision Support Tool
recommended action and related risk
GIS server
maritime operation under consideration GIS client taken action
spatial map with recommended actions and related risks Decision Authority
spatial maps with underlying environmental forecasts
Figure 1: The FR-TDA system implemented at CMRE and its core: the DST (dashed box). The system takes the forecasts of the environmental variables affecting a predefined operation (amphibious landing, glider recovery, etc.) as input, and provides a spatial map with the recommended action (and the related risk level) for each point in the space inside a pre-defined bounding box. The DA can scroll maps over time, and select the best time instant to perform a particular action
-4-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
2.2 The Decision Support Tool The DST is based on a Bayesian risk assessment framework. In particular, minimum risk decision theory [4] is used to build a model for evaluating the risk related to the future state of the environment for the considered maritime operation, thus helping the DA to take appropriate actions in response to such forecasted environmental state. The framework has to provide a realistic evaluation of the risk in terms of costs of taking a particular action (e.g., run mission, reschedule mission) under specific future environmental conditions for the considered maritime operation. The Bayesian risk assessment framework assumes that the overall environmental condition is categorized in a pre-defined number of classes. In this paper, we have considered only three classes: favourable, marginal, and unfavourable. A favourable environmental condition is associated with a future state that encourages maritime operations. For instance, for glider surfacing and recovery operation a favourable condition means that all the forecasted environmental variables which affect the operation itself (Significant Wave Height, Wind Speed, Sea Current and Vessel Traffic Density) are low. On the contrary an unfavourable environmental condition is associated either with at least one of them very high, or most of them quite high. All situations in the middle should be tagged as marginal. The Bayesian risk assessment framework requires posterior probabilities associated with each class (coarsely speaking, something like this: favourable: highly probable, marginal: low probable, unfavourable: low probable). Once such posteriors are available (they will be predicted by a fuzzy classifier which uses the environmental forecasts as input), the risk assessment framework has to provide a measure of the risk associated with each action. Such information is then used to rank actions and to allow selecting the action associated with the minimum risk. Figure 2 shows the architecture of the DST. It is made up of three components: a fuzzy rule-based classifier, a risk calculator and a minimum-risk action selector. The following sections will describe those blocks in details. environmental forecasts (F)
fuzzy classifier
posterior probabilities associated with each environmental condition class (M)
KB
domain expert(s) knowledge elicitation according to the considered maritime operation
risk calculator
class/action cost matrix
risks related to each action (N)
minimum risk-based action selector
selected action and related risk
Figure 2: The schematic of the DST. It is made up of three components: a fuzzy rulebased classifier, a risk calculator and a minimum-risk action selector. The fuzzy rulebased classifier is used to compute the posterior probabilities associated with each of the M environmental condition class, exploiting a Knowledge Base (KB) and the forecast of the F environmental variables. The risk calculator computes a risk for each of the N available actions, using a Bayesian model and a class/action cost matrix. -5-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
2.2.1 The fuzzy rule-based classifier The fuzzy rule-based classifier is the first block constituting the DST (see figure 2). Like statistical classifiers, it receives the environmental forecasts as input and provides the posterior probabilities for each class P m x where m is the mth class label with m=1,...,M and x lR F is the vector of input METOC variables. A Fuzzy Rule-Based Classifier (FRBC) [5] is made up of two parts: the Knowledge Base (KB) and the Fuzzy Reasoning Method (FRM). The KB, in turn, is made of a Data Base (DB) and a Rule Base (RB). The DB contains the information about the fuzzy sets used to partition each variable (i.e., the parameters associated with the membership functions which describe a fuzzy set). The fuzzy partition defined on the f th variable is f { A1f ,..., AQf } , where Aqf f
is the qth fuzzy set over the f th variable X f and Q f the number of the fuzzy sets employed. As regards the RB, here we consider to have an RB Rk k 1 made of K if-then rules, each K
one having the following form: Rk : If X 1 is A1jk ,1 and ... X F is AFjk ,F then X ( F 1) is A(jkF,(F1)1) , ..., X ( F M ) is A(jkF,(F MM ))
(1)
where X 1 ,..., X F are the input linguistic variables, X ( F 1) ,..., X ( F M ) the M output linguistic variables, and Ajf means that jkth, f fuzzy set is used on the f th variable, k,f
f 1,..., ( F M ) .
So doing, the whole RB can be stored in the following J matrix: j1,1 J j K ,1
j1, F
j1,( F 1)
jK , F
jK ,( F 1)
j1,( F M ) . jK ,( F M )
(2)
For example, let us consider a maritime operation affected by two variables only (e.g., wind speed and wave height), where the number of environmental condition classes is three (favourable, marginal, unfavourable), consisting of four rules (K = 4) and a number of fuzzy sets for each input and output partitions equal to three ( Q f Q 3 ). Under these settings, a consistent J matrix would be the following (1=Low, 2=Medium, 3=High): 1 2 J 1 3
3 3 1 1 1 1 1 3 . 2 1 3 1 3 1 2 3
(3)
-6-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
2.2.2 Rule Base generation and fuzzy reasoning method There exist many ways to design the RB. Mainly, we can distinguish between knowledgedriven and data-driven approaches. Our system supports knowledge-driven approach, which means that it allows the user to define both the data base (i.e., the membership function values) and the rule base. This approach is interesting because it allows the system to integrate past experience of human expertise of each operation at sea, and to run the system even when few or no historical data exists. Currently, the data-driven approach is not supported by the system (though we are investigating its implementation). It may add value in cases when a statistical significant amount of data is available and thus an automatic rule based can be generated without human intervention. Under the knowledge-driven approach, the user is asked to provide a data base and a rule base, both consistent with the maritime operation under consideration. Once done, the system is able to perform the classification. The use of a fuzzy rule base instead of a nonfuzzy rule base allows a greater flexibility in defining the data base, instead of using fixed thresholds on input forecasts. As regards the fuzzy reasoning method, we have used a First Inference Then Aggregate (FITA) approach based on the singleton fuzzification method (i.e., we have used real numbers as inputs of our fuzzy system) and the Mamdani inference method [6]. More precisely, the minimum has been used to model the AND operator in if-then rules. The same operator has been used to model the implication for deducing the consequents of each rule. The output fuzzy sets obtained after implication have been aggregated using the maximum, while the posterior probability associated with each output class has been computed as the centre-of-gravity method.
2.2.3 The risk calculator This unit computes the risks related to each available action, based on the posterior probabilities associated with each environmental condition class and the costs related to each pair (environmental conditions class, action). Classically a minimum risk model can be formalised in the following way. Suppose the environmental forecasts (also called risk factors) for a determined position and instant in T
t (i and j are spatial indexes on a time are arranged in a column vector xtij xijt 1 ,, xijF regular grid, while t is the temporal index) and that there is a set of M environmental 1 ,, M ( condition classes
1 favourable , 2 unfavourable, 3 marginal in our case) having a set of
related posterior probabilities P m xijt , calculated by means of the fuzzy rule-based classifier (see previous section). Under a particular class m , an action n may be taken among
the
N
actions
in
the
-7-
set
A 1 ,, N ,
(
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
A 1 run mission , 2 reschedule mission in our case, i.e., N=2) with a cost
n m . The costs can be arranged in the following cost matrix 1 1 1 2 2 1 2 2 N 1 N 2
:
1 M
2 M N M
(4)
Each row of the matrix represents an action and each column an environmental condition class. The average risk for each action over the set of classes is defined as:
r n xtij
m
n
m P m xijt ,
(5)
and in matrix notation as:
rijt ptij
(6)
T
T
where rijt r 1 xtij , , r N xtij and p tij P 1 x tij , , P M x tij .
Thus the actions are ranked according to the value of the corresponding risk and the action with the minimum risk is the one that guaranties the safest action under the predicted environmental condition, which in turn is likely to occur on the area of interest at the considered instant of time in the future. Clearly, the cost matrix has to be provided by human experts of the maritime operation under consideration. In particular, under our assumptions, is a 2 by 3 matrix: 1 1 1 2 1 3 . 2 1 2 2 2 3
(7)
The costs have to be set according to the considered maritime operation and the involved assets. Typically, for action 1 (run mission) costs grow as the condition becomes more unsafe; the opposite happens for costs related to action 2 (reschedule mission).
2.2.4 Uncertainty propagation The risk calculator can be considered as a non-linear random vector mapping between the t F and the action conditional risks, input space of METOC factors, xij
-8-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
rijt [r (1 xtij ), , r ( N xtij )]T L . The unscented transform (UT) is the statistical t
tool that is used to propagate the statistics of xij through the risk calculator up to the t
second order. In particular, the mean METOC factor vector, xij , and the covariance t
matrix, Xij , predicted by a SE model, for instance, are the inputs to the UT wrapping the risk calculator. The UT is an efficient way to propagate second order statistics (mean, covariance and cross-covariance) of a random variable passing through a non-linearity of any order [2]. The UT estimates the statistics of the output random vector knowing the statistics of the input by approximating the true probability density function of the input with a Gaussian and making use of a deterministic sampling procedure that is more efficient than classical random sampling Montecarlo techniques. Advanced versions of the technique use information on higher order moments of the input distribution, resulting in an improved estimate of mean and covariance of the output [2]. These features will be used in future work for those cases in which the input presents highly non-Gaussian characteristics. The output of the risk calculator is evaluated for a set of so called sigma points:
t sptijk wk , ijk
with
i = 1,…,2F + 1
(8)
ijkt F . The UT deterministic sampling scheme generates the samples ijkt in the
input space by using the METOC statistics and the weights wk . The same weights and the risk calculator outputs evaluated for each ijk are then used to estimate the statistics of t
the output by means of sample weighted averages as reported in Appendix 3. As specified by Eq. 3, the UT makes use of S=2F+1 sigma points to estimate the output statistics (in case of 4 inputs, as in the present work, the number of sigma points is just 9). In those cases for which only the input statistics are available, we need a method to resample and propagate the uncertainty through the system. In case of a large ensemble with Ne>S (Ne being the number of members of the ensemble), calculating the ensemble statistics and then using the UT is more efficient than providing the ensemble members directly to the DSS system and then estimating the output uncertainty. Moreover, the data transfer from a remote METOC centre would benefit if only the statistics are transmitted through the network. Furthermore, in the case where an SE model with assimilated measurements is used, propagating each single model used in the SE and then estimating the output uncertainty is different than propagating the statistics of the SE which are a combination of models and measurements. This paper presents the system taking into account the more general scenario even though the number of models available to the SE is less than S and no measurements can be assimilated into the SE. The DSS concept provided in this work is valid in any case, independently of the quality of the input statistics. In case of poor quality statistics, the operator may still be interested, for example, in performing a what-if or sensitivity analysis by testing different levels of uncertainty and bias to compare different future scenarios. The user can either trust the
-9-
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
system if the scenarios are not significantly different, for example, or decide on the basis of experience and constraints of a different nature, beyond those considered in the DSS. t T The estimated risk mean vector, rijt [r (1 xtij ), , r ( N xtij )]T [rijt1 ,..., rijN ] , and the
t
risk covariance matrix, Rij , at the output of the UT (recall that for an action list of two actions, which is the case examined in this work, the risk mean vector is 2x1 while the covariance matrix is 2x2) are used in the action selection and confidence level calculation steps as described in the following sub-section.
2.2.5 The minimum risk action selector and decision confidence The system chooses, according to the classical Bayesian decision theory [4], the action having the minimum conditional risk:
ˆijt arg min[r ( l xtij )] ,
(9)
l A
This step is postponed, as proposed in [7], until the UT (see Fig. 2) in order to propagate the input uncertainty as described in the previous section. The final architecture is then able to exploit information from multiple METOC centres and the uncertainty associated with METOC forecasts so as to improve the situational awareness (SA) of the mission planners. The final decision law is obtained by substituting r ( l xtij ) in (4) with the mean risk vector components, r ( l xtij ) rijlt , at the output of the UT:
ˆ ijt arg min[ r ( l xtij )] .
(10)
l A
The decision confidence step detects those cases in which the actions are not statistically separable as a consequence of too much uncertainty in the METOC forecast inputs. As t proposed in [7], the system estimates a decision confidence measure, cmij , by comparing the 95% action risk confidence intervals, which are calculated by using the diagonal t elements of the risk covariance matrix, Rij and the mean vector, rijt , both estimated by t
the UT. The confidence measure cmij is a heuristic index which was inspired by methods like Tukey’s multiple comparisons procedure or the Gabriel’s test, which find application in testing if several groups of samples belong to the same statistical population [8][9]. Future developments of the system will improve the confidence measure by considering a t t formal hypothesis test procedure and correlation among risks. If rijUp rij 1 1 is
the upper limit of the confidence interval associated to the chosen action and
rijt Lo min rijt p p p 1
is the minimum of the lower limits associated to the
- 10 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
remaining actions (p=1,…,N and p is the permutation of the action indexes, l=1,…,N, induced by the rank of the mean risks in ascending order), the difference t cmijt rijUp rijt Lo or a normalised value of it (for example by the square root of the sum of risk variances) is the confidence given to the decision. Confidence to the selected t action is given for positive values of cmij , meaning that the confidence intervals of the t
action risks are not overlapped. In addition, a binary confidence map, cij , is produced as t
well by detecting negative and positive values of cmij : t 0 cmij 0 cijt , t 1 cmij 0
(11)
helping the mission planner to easily locate confident and non-confident decisions in space and time. In this way, the system provides an additional level of decision, the t confidence map, cij , to inform the user that the system itself is not able to make a final decision due to statistically contradicting information. The user, in these cases, can either accept the proposed action or distrust the system and decide on the basis of his experience, different criteria and the operational context.
- 11 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
3
NICOP: Tasks 1 and 2 The goal of tasks 1 and 2 of the NICOP is to review and evaluate the FR-TDA system interface and make recommendations for improving the interface that could support operator decision making in future maritime operations. Specifically, task 1 and 2 of the NICOP include: T1. Review of the CMRE FR-TDA application and products. T2. Specify future design lines to follow in order to improve the interface and the products presentation.
3.1 Background: Scope, Limitations and Assumptions This section of the NICOP proposal addresses the first two tasks related to the current CMRE FR-TDA interface design. Namely, the goals set out in the proposal were to evaluate the fuzzy rule-based DSS system and make recommendations to improve the interface and its future design. The original proposal, focused on developing a knowledge elicitation task to collect data from METOC SMEs. We addressed these tasks from a Human Factors (HF) analytical perspective. The original goals were to provide the best support to the mission planner by: a) Knowledge elicitation of both system rule bases and utility functions from domain experts in a rigorous scientific way so as to represent with a certain degree of confidence the uncertainty in the operational constraints and the degree of importance of input variables, and to avoid system biases. b) Design of GIS system interfaces and assessment of their effectiveness and suitability, with respect to the classical METOC impact matrix traffic light visualization, in reducing operator cognitive burden. CMRE is actively involved in applied research on fuzzy rule-based Decision Support Systems (DSS) culminating in the development of a prototype system which integrates interoperable Web GIS services with a fuzzy rule-based inference engine into a common GIS interface application as outlined in chapter 2. Specific human factor competences are needed to address the issues presented in this proposal. Thus, METOC information analyses, hypotheses testing and mission planning during glider operations using the GO-DSS system will play a pivotal role in CMRE future exercises and experiments. These events will provide an opportunity for CMRE to gather - 12 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
meaningful data regarding the utility and impact of the GO-DSS system on operator decision making and mission planning. Further, experiments at sea will also provide an opportunity to gather feedback from the users regarding the systems effectiveness in evaluating risks, confidence levels, cost/benefit assessments, and the impact of interface design, as well as make recommendations for key features that may be integrated in future designs of the GO-DSS system.
3.2 Introduction Military and civilian maritime operations require an understanding of the impact of environmental factors, i.e., meteorological and oceanographic conditions on human and system performance, especially those involving assets such as gliders and unmanned systems, which may put both people and systems at safety risk. To be successful in the dynamic operational environment, one must evaluate the maritime environment and enhance the decision maker’s ability to achieve SA for more effective mission planning. Specifically, it is essential to develop tools that will provide a means of accurately assessing the impact of weather on maritime operations, while reducing risks associated with mission performance [1]. The aim of providing the user with an intuitive visual display will assist them in achieving SA [8] and understanding of the environment is critical for the development of operational mission planning. The FR-TDA system was developed to provide the user with the means to evaluate the impact of environmental (e.g., wind velocity, current speed, wave height, etc.) and nonenvironmental information (e.g., traffic density) on mission planning for maritime operations. The FR-TDA system interface provides the user with information that enables him to evaluate variables provided by multiple forecasting centres and models. However, there is a great deal of uncertainty in the data itself which the FR-TDA system processes to yield a traffic light map that provides the user with an intuitive graphic display and predictive pathways for safer maritime operations. For this discussion, we will focus on glider operations. However, it is worth noting that the GO-DSS framework in general and in particular the FR-TDA system would be equally applicable and useful for planning the deployment of sonobuoys, for counter piracy missions, ASW, Maritime Situation Awareness (MSA) and Mine Counter Measures (MCM) missions.
3.3 Research Objectives: Review of the CMRE FR-TDA system The goal of the FR-TDA application is to develop an integrated system that will increase the automation and effectiveness of information fusion, environmental impact and uncertainty assessment and enhance the operator’s decision making processes. The FR-TDA system makes use of a fuzzy logic inference engine with web enabled geographic information systems (GISs) which provides the operator / decision maker with
- 13 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
the capability to evaluate numerous factors and hypotheses during the course of mission planning for maritime operations. Thus, the purpose of the system is to enable the operator to make timely decisions within a geo-spatial context. There are a number of challenges in the field of decision making under conditions of uncertainty, such as mission planning and selecting an optimal course of action in METOC operations where the data is uncertain and often conflicting and/or contradictory due to differences in models, data scarcity, and/or poorly designed sensor networks. The traditional approach to developing an interface is to conduct cognitive work analyses in association with SMEs to determine the strategies employed during their decision making process. Due to constraints beyond the control of the investigators, i.e., lack of an Institutional Review Board (IRB) protocol, we are limited to review the interface within the framework of the GO-DSS itself and evaluate how each task is conducted, identify sources of METOC data and how these are evaluated, how hypotheses are developed and make recommendations to enhance the interface design and METOC operator’s ability to make tactical assessments and develop mission plans, based on the output of the application displays. The technical approach for this analysis included an evaluation of various sources of METOC data input, and principles of good practice in visualization based on User Centred Design principles to develop a visualization interface display.
3.4 Technical Approach: Evaluation of the Interface Design This study is aimed at addressing ways to improve the effectiveness of the FR-TDA system by examining ways to exploit METOC data and present information that will help support the METOC decision maker as they form hypotheses and assess risks involved in mission planning for the deployment of gliders. Specifically, the goal is to assess human factors in the GO-DSS decision support system that uses METOC data. The FR-TDA system Web-GIS services include, as illustrated in Figure 1, a geospatial catalogue to search for available forecast models in time and space and a map server for METOC grids and FR-TDA outputs retrieval and displaying. The current FR-TDA system interface display illustrates the traffic light mapping that intuitively guides the decision maker in their evaluation of the contributing variables. User-centred analyses have shown the benefits of affording the operator an opportunity to interact with the data [9]. Interactions with the interface afford the user an opportunity to explore relationships among variables which facilitate the development of their mental model and achieve an understanding of the operational environment [10].
- 14 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
a)
b)
c) Figure 3: Fuzzy rule based Web-GIS DSS application. a) Web METOC catalog. b) Web service for METOC retrieval and displaying. c) Fuzzy rule based geo-spatial decision making and displaying.
- 15 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Two dimensional (2D) visual search interfaces are fairly straightforward in their design and provide graphic representations of relationships among variables. For this evaluation, environmental (meteorological and oceanographic) data input is forecasted by various models forming an ensemble which are taken as input into the system and uncertainties associated with differences between models are resolved using statistical tools. In this way, the system provides a traffic light map (Figure 1-c) which affords the user an intuitive decision support system (note that this example is binary red-green whereas traffic light would typically be red-amber-green to take into account marginal situations). The goal of this assessment is to evaluate the effectiveness of the interface and determine if there are areas for enhancing the current interface design. Advances in modelling and computational power have allowed for an increase in the volume of data that can be provided but not necessarily managed effectively by the user. The challenge here is to determine how to effectively exploit the information, while ensuring that they can query the system, challenge hypotheses and increase their confidence levels during the course of their decision making. The technical approach taken for this task was to review the source and data input processes, achieve an understanding of the ways in which the operator can work with the raw data and identify areas for enhancement of the visual display and make recommendations for future developments of the interface design. The traditional approach to address this problem is to apply User-Centred Design principles which include conducting laboratory research experiments with users of the system, e.g., METOC operators. The decision was made to develop the tools (e.g., surveys, questionnaires, etc.) that would be distributed to METOC operators during CMRE exercises and experiments. Each operator would be asked to complete the forms as they used the FR_TDA system to develop mission plans. The questionnaires were aimed at gathering critical insights regarding the ways in which METOC operators use the system and achieve a level of SA that informs and guides their mission planning and decision making. From a system designer’s perspective, usability is a critical component of assessing the effectiveness of the design itself. There are risks and costs associated with the time and effort in using a system that fails to support the decision maker. An essential principle for optimizing the design of a system is to evaluate the performance of the system with SMEs. There are also alternative methods to include user testing on system designs, such as a usability audit. This approach can be conducted with SMEs and will also yield critical insights into the success or failure points of a prototype. The analytic/evaluation of a system such as the FR-TDA could therefore be undertaken using the usability audit approach to incorporate user feedback to optimize the application design. Human performance is a function of cognitive processes such as perception, attention, memory and the ability to evaluate alternative hypotheses and make decisions. The field of cognitive engineering provides a framework for evaluating human performance and decision making using an interdisciplinary approach that includes various tools and techniques for evaluating human performance [9].
- 16 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
The basic approach of the user-centred system design includes an evaluation of the interface to address its effectiveness for operator interactions. Namely, does the system support the decision maker’s goals and task objectives? How does the system reflect the range of tasks that need to be performed by the domain expert? What strategies does the expert use to achieve their objectives? These are some of the relevant questions that need to be addressed during the evaluation of the system. A number of principles must be considered when evaluating the design of an interface. Among these, consideration must be given to the specific domain tasks to be performed by the user, such as METOC operators developing mission plans under conditions of uncertainty. Given the complexities of the environmental information (i.e., meteorological and oceanographic data) and non-environmental information (i.e., traffic density), the interface design should provide a means of simplifying tasks to reduce operator cognitive workload and enhance an operator’s decision making. For this study, we considered the way the operator used the GO-DSS system to gather relevant information that was essential for developing their mission plans. For METOC operators, the uncertainty associated with environmental variables presents significant challenges to developing an accurate and effective mission plan. For example, METOC operators must develop an optimal course of action for gliders to be deployed safely and effectively. To this end, they must consider several of meteorological and oceanographic variables, such as wind velocity, current speed and wave height in their evaluation. In addition, they must understand the operational context and the maritime traffic in the region, in order to achieve SA for deploying gliders effectively and safely. The FR-TDA system integrates numerous variables and operational rules through a fuzzy logic inference engine in order to reduce the information load and to provide the user with products that synthetically describing the situation in the operational field. It is worthwhile to mention that the optimal course of action is being evaluated in terms of risk factors for the gliders in order to take into account the safety part of the deployment plan. In future works, the optimal sampling strategy (such as adaptive sampling methods) has to be included in the planning process in order to optimize the effectiveness of the operation. The FR-TDA system was used in the CMRE Recognized Environmental Picture (REP 10) experiment which took place in the Ligurian Sea from 19 August to 3 September 2010. The sea trial was focused on the exploitation of a variety of observational assets, including remote sensing satellites, underwater gliders, etc. For this discussion, it is worth noting that the system provides an assimilation of METOC models and METOC data collection. The uncertainty associated with METOC measurements and forecasts was evaluated and integrated as part of the system to provide outcomes to the METOC operators aimed at improving their decision making and ability to conduct mission planning. The question raised is, can the FR-TDA system be improved? What are the critical gaps, if any, in the current system design? There are a variety of issues which emerge as METOC operators make decisions under conditions of uncertainty. For example, how do METOC operators evaluate each variable and its contribution to the overall perspective of the mission? What weighting do they assign to each environmental variable and how does this influence their decision
- 17 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
making within the operational environment? How do they evaluate the validity of the data? How do they evaluate different hypotheses and evaluate risks? What is the right level of information to be provided to the METOC operator? One approach to managing vast amounts of data to be presented to the end user is to use visual analytics. Visual analytics provides a means of representing the complexities of the data in a meaningful manner that will facilitate the METOC operator’s ability to interpret vast amounts of data and gain insight into the inter-dependencies of environmental factors that change over time.
3.5 Visual Analytics and Implications for Interface Design Visual analytics integrates computational analysis techniques with visual representations affording the designer a means of developing a visual display that is both flexible and intuitive. It also provides a means for an operator to achieve an understanding of complex data sets and make informed decisions. Visual analytics is often used to integrate computational models, such as fuzzy logic rule-based models, and to illustrate complex relationships among variables, such as those evidenced in METOC models. For example, METOC operators must monitor complex meteorological and oceanographic variables (i.e., wind velocity, current speed, wave height, etc.) as they develop their mission plans for maritime operations. Often this data is gathered from a distributed network of sensors and often requires vast amounts of data to be analysed and processed. Thus, the visual display and its graphic representations illustrate a complex network of relationships that allows the user to interpret vast amounts of data and gain insight into the dependencies of each of the variables and changes over time that would otherwise not be readily available. Given that the relationship among METOC variables is spatiotemporal and graphically displayed in the GO-DSS system, the question arises as to what relationships are not evidenced in this graphic display? What, if any, information may be lost in translation during this process?
- 18 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
*User Interactions
Figure 4: FR-TDA system feedback Loop: User Interacts with Data to Acquire Knowledge.
3.6 Conclusions & Recommendations The FR-TDA system provides an effective visual display of the contributing variables. However, upon closer evaluation it is possible to identify several potential shortfalls in the design. Among these, there is a need for the user to be able to interact with the data and get feedback regarding the relationships among variables. There is a great deal of uncertainty and risk associated with METOC data. Therefore, METOC operators must be able to explore the data in order to make an informed decision under conditions of uncertainty. As data sources become more complex, there is a critical need to provide METOC operators with the ability to achieve an understanding of the spatiotemporal relationships among variables and how these relationships may change over time. The ability to evaluate temporal relationships among meteorological and oceanographic variables is especially critical when considering developing mission plans within a 72 hour a priori time window. The METOC operator must interact with the data to identify anomalies, evaluate different models and associated risks, and to evaluate alternative hypotheses as they come to understand the operational context of their proposed mission plans. The METOC officer who is analysing data and generating a mission plan a priori would benefit greatly from being able to explore and evaluate the data by isolating specific variables of interest (e.g., wind velocity, current speed, etc.), identify parameters of atmospheric and/or oceanographic conditions that are anomalous and determine the - 19 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
potential impact of these conditions on the mission itself. This approach would also enable the METOC decision maker to evaluate risks associated with each of the variables. At the present time, the FR-TDA system does not provide the operator with
direct feedback nor the ability to evaluate spatiotemporal relationships and their associated variables independently,
risk metrics and confidence level metrics that are intuitive to the user,
alert algorithms to cue operators to explore a specific region further.
Rather, the system presents solutions to the user which may potentially mask or statistically minimize the impact of critical information for the METOC operator to make an accurate, informed decision. It would be helpful if the system allowed the user to
interact with the data,
test various hypotheses and also to alert the user to areas of critical concern that should be explored further prior to making a final decision on mission plans.
The survey questionnaires (Appendix A) completed by METOC operators will provide a means of evaluating the METOC operator’s perspective on gathering data input using the GO-DSS system. Further, we contend that the effectiveness of the GO-DSS system for mission planning will be related to the success of glider operations during CMRE experiments and exercises.
- 20 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
4
NICOP: Task 3 Tasks three and four are inter-related in that each makes reference to the development of a research protocol for conducting a knowledge elicitation study to explore the ways in which SMEs use the FR-TDA system and how the system might be improved.
4.1 Background: Scope, Limitations and Assumptions This section of the research undertaken via the NICOP addresses two tasks related to the development of a research protocol for enhancing the current design of the GO-DSS interface design. The goals set out in the original proposal were to develop the research protocol for knowledge elicitation of Subject Matter Experts (SME) using the GO-DSS system in order to evaluate its effectiveness and identify potential improvements. There are several traditional approaches to evaluating visual interface designs, including conducting a user-centred design study.
SME Display Interaction
Data Analysis
Enhance Design
Figure 5: User-centred design analysis paradigm
User-centred design analysis (Figure 5) affords the human factors designer the opportunity to evaluate the human interaction with data visualized on an interface. In the field of visual analytics, wherein complex spatiotemporal data is presented, human interaction with the data is critical to achieving an understanding of how the decision maker makes sense of the data, attains SA [8] and makes effective decisions. Although the constraints related to the IRB restrictions precluded our ability to conduct the study, we did decide to design the research protocol for upcoming CMRE experiments. - 21 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
4.2 Introduction Visual displays of METOC spatiotemporal data are intended to convey complex relationships among a number of variables such as wind velocity, temperature, wave height, etc. Military operational planning requires METOC forecasts to ensure safe and successful operations. Often, as is the case of demonstrating unmanned systems for Rapid Environmental Assessment (REA) in a NATO exercise and CMRE experiments, the operation will require the deployment of autonomous unmanned vehicles. Thus, the need to deploy and recover assets in a safe manner is a critical component to any military mission. It is therefore essential for the METOC operator to be able to forecast the environmental conditions to ensure the optimal planning for such events. However, as has been well documented, METOC data is replete with uncertainty and the need to represent uncertainty and understand the environmental conditions for such tasks remains a challenge. Tools such as the FR-TDA system should therefore play a vital role in the successful planning of such missions. Given the complexity of METOC data, it is therefore essential to examine ways to improve the design of the FR-TDA system to ensure that the operator can achieve an accurate assessment of the information contained within the visual display. Often, the most important information is embedded in an algorithm and is irretrievable to the operator to examine in detail. Therefore, one of the critical concerns in developing an improved visual display is to think about the user and how we might address their requirements for extracting information from the system.
4.3 Technical Approach The previous section provided a description of the interface design layout in some detail. For this section, we will focus on the development of the research protocol and the design of the technical approach to address questions regarding user requirements for extracting information from the system, how to identify what the critical information requirements are, and finally how to determine those requirements that will enhance the development of the FR-TDA system design. The traditional systems engineering design approach to information processing system design is to first develop the system and then test it on the user. In contrast, it is highly recommended that Subject Matter Expert input be solicited throughout the earliest stages of the design process to ensure that their requirements for information management have incorporated. The design of any visual interface display that represents complex spatiotemporal data benefits with input from SMEs and reduces the potential for error in the system design. Designers observe SMEs and learn how they interact with the data and identify critical characteristics and interpret information conveyed in the visual display. Designers can observe SMEs as they explore the data to shape their understanding of the environment during their decision making process. For this discussion, we are referring to the ways in which (METOC) SMEs interpret the information on the GO-DSS visual display to understand the environment and its constraints for mission planning. However, the same rules would apply across all domains. Experts extrapolate critical cues, data and build relationships as they develop their mental models based on their interactions with the information conveyed in the interface display.
- 22 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
The impact of visual analytics in conveying complex information in spatiotemporal displays has already been discussed in some detail. However, one must consider both the human’s cognitive and perceptual processes for analyzing information presented on visual displays, as well as the role of the Subject Matter’s Expert’s training, education and experience in this regard. The cognitive and perceptual processes of the human are well documented. However, it is worth noting that there are significant quantitative and qualitative differences among SMEs. For the METOC operator, there are unique differences regarding the METOC modelling tools, environments in which they have had experience, and the recency of their experiences in a specific domain. Thus, each SME has a unique history with regard to the level of expertise, quality and quantity of training, as well as the recency and quality of experience. Each of these characteristics of the human operator tells us something about the level of expertise the SME brings to the task and how they might approach the information from a unique perspective based on their experience, training and expertise. In addition, one must also consider the cognitive constraints related to workload (i.e., how much information can an individual process during a limited time, fatigue effects, stress, etc.) and potential cognitive biases. Some researchers have focused on the characteristics of the system itself, ignoring the human’s interactions. However, evidence has shown the value of considering the relationship between the task, the environment and the user [11]. The Ecological Interface Design (EID) approach also suggests that we consider human cognitive and perceptual processes and the ways in which users interact with the system [12]. This EID framework suggests that the designer seek to optimize human performance by exploiting the capabilities of human perceptual processes during the design process. For the GODSS designer, one could exploit the relationship between an individual’s perception of the environment and actions that could be taken. This approach is especially important for the task of the METOC operator who must assimilate environmental variables to develop a mental model of the environment as they develop their mission forecasts. Other researchers, such as [13] have drawn a distinction between focused attention tasks and more complex tasks. Their principle postulates that complex, high level tasks, such as METOC mission planning, require the use of more integrated displays such as the FRTDA to be effective system. However, to evaluate the effectiveness of the FR-TDA system, it is essential to conduct the cognitive task analysis [14], as originally proposed in the NICOP. The objective of the NICOP was to develop an integrated GO-DSS prototype to increase the automation and effectiveness of information fusion, environmental impact and uncertainty assessment, and decision making processes. To this end, it was planned to conduct cognitive work analyses via observation and interviews with SMEs using the Applied Cognitive Task Analysis (ACTA) method. ACTA is a set of knowledge elicitation and representation techniques intended to assist in identifying the key cognitive elements required to perform a task proficiently [14]. Questionnaires have to be distributed and be completed by METOC operators as they develop their mission plans. The knowledge audit portion of the study will focus on the level of expertise required to complete the tasks. For example, the questionnaires will request information regarding their level of experience, knowledge and expertise. In
- 23 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
addition, comments will be elicited to address the question as to how SMEs use and/or rely on specific cues and strategies for developing mission plans. As aspects of expertise are elicited they are probed for further details and concrete examples (e.g., in this situation, How do you evaluate risk with regard to the variables presented in the display? What strategies are you relying on? ). It is this ability to probe a variety of knowledge types (e.g., perceptual skills; cognitive processes and data anomalies) that provides a more complete description of cognitive processes and sets the ‘knowledge audit’ aside from structured interview and task analysis techniques. Lastly, interviews will be conducted with SMEs to better understand their approach to data represented in the FR-TDA system visual display. The results of these analyses will be merged into a table to establish an understanding of the cognitive demands and used to focus the analysis and identification of common themes in the data. Furthermore, the results of these analyses will be used to develop requirements for the design of an enhanced system.
- 24 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
5
NICOP: Task 4
Task T4 outlines a system evaluation protocol that involves the participation of potential system users. The following discussion is the proposed research protocol to be completed during NATO exercises and CMRE experimentations. The objective of the proposed study is to conduct Human Factors (HF) analyses of operators using the FR-TDA system. Specifically, this research protocol is designed to collect data from SMEs who will be using the system to develop METOC mission plans for operations at sea such as glider operations being conducted as part of a CMRE experiment or NATO exercise. The results of this study will be used to develop requirements for the development of an enhanced tactical decision aid.
5.1 Technical Approach METOC operators will be asked to complete questionnaires as they develop mission plans for the deployment of gliders during an exercise or experiment. Given the complexities and uncertainties associated with METOC environmental data (i.e., meteorological and oceanographic data), questionnaires will probe each Subject Matter’s expertise and strategy as they evaluate environmental variables, assess risks, and develop mission plans under conditions of uncertainty. The goal is to achieve an understanding of critical cues they might use and gain an understanding of their approach to developing a mission plan by evaluating complex information in a dynamic environment. Questions raised include, how do they evaluate risk, assess the validity and reliability of data throughout the mission and how well does the FR-TDA system support their decision making strategies. METOC operators make decisions under conditions of uncertainty by exploring the data (e.g., wave height, wind velocity, etc.) and developing Situation Awareness [8] and an understanding of the environment. For this study, METOC operators will also update their mission plans in order to sustain safe operations for both glider assets and for the people involved in the recovery of the gliders upon completion of the exercise. Questionnaires will be provided to them to assess their adaptation to change throughout the exercise. This study is focused on evaluating the benefits and shortfalls of the FR-TDA design and to establish how METOC operators interact with the system. We contend that there is a need to address shortfalls in the system design, such as feedback to the user, graphical representations of risk assessments, and the operator’s ability to interact with the data. - 25 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Currently, there are often competing if not conflicting mission objectives that may negatively impact decision making and planning. We contend that gathering information from as many METOC operators as possible during the course of an exercise will afford the enhancement of the FR-TDA system. It is worth noting that if this were a laboratory study, it would be worth conducting the study with both novices and expert METOC operators in order to evaluate the different approaches to the data, assess the impact of experience and expertise, as well as to distinguish if there are any critical cues we might garner from the expert’s approach to interacting with the METOC data. However, given the constraints of the IRB protocol, we are unable to address these questions in a formal laboratory study at this time. We anticipate that the results of this study will yield insights regarding the strategies employed by METOC operators as they develop mission plans for the maritime environment. The results of this study will be used to develop requirements for an enhanced GO-DSS tactical decision support system. We contend that the integration of feedback mechanisms, risk assessment metrics, and measures for evaluating the validity and reliability of the data will greatly enhance the system capabilities and thereby facilitate optimal METOC operator performance and decision making.
5.2 Conclusions and Recommendations The FR-TDA system provides a valuable tool for METOC operator to conduct mission planning under conditions of uncertainty. The fuzzy rule-based model embedded in the system design provides the user with an ability to assess the meteorological and oceanographic environment. The system facilitates better decision making and improves overall human performance. However, there is always room for improving the design by analysing shortfalls in the system during operational performance. Among these areas, we are exploring ways to incorporate operator feedback to evaluate variables and provide metrics for risk assessment. We contend that the capabilities provided by the system should improve human performance and increase an operator’s ability to manage complex data. For the future design of the system, we will utilize the results of future exercises to integrate METOC expert requirements for decision making and incorporate the ability to explore and exploit data throughout the decision making process. It is challenging to make recommendations based on a study that has yet to be completed. Perhaps, it might be best to attempt a few hypotheses of anticipated results. We anticipate that METOC operators with significant at-sea experience will use the system effectively. We contend that there is a benefit for providing the user with feedback and the ability to explore and exploit METOC data. Further, we anticipate that METOC operators will provide that feedback to us following the exercise. Lastly, we highly recommend that a laboratory study be conducted as it will yield critical cues that might otherwise remain undiscovered. Our recommendation includes that we continue to pursue conducting a laboratory study to support the future development of an advanced FR-TDA system. We contend that the system has significant implications for the decision of future tactical decision aids such as those used in the Maritime Surveillance program. - 26 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
6
NICOP: Task 5.
6.1 Conclusions and Recommendations The experimental design and technical approach recommended for this study include knowledge elicitation, cognitive workflow analysis, cognitive task analysis and usercentred analysis. Each of these methods serve as an established means to identify potential areas for improving system design and enhancing operator interactions and efficiency, reducing errors and minimizing cognitive workload in system design. The experimental research protocol and supporting documentation and questionnaires to conduct Task 5 were completed by Dr. Masakowski during her tenure at NATO CMRE. Specifically, analytical approaches were outlined to include knowledge elicitation, cognitive task and workflow analysis, as well as a user-centred design analysis were developed, questionnaires were designed and the IRB protocol was developed and submitted. Further, the experimental design was developed for application in future NATO CMRE experiments. Although the experiment developed for Task 5 could not be conducted by a US researcher (SECNAV Instruction 3900.39D), further study may be conducted by a NATO CMRE researcher and is highly recommended. The human factors analysis, techniques and tools established for this study could be utilized by a NATO CMRE researcher as a means of evaluating the effectiveness of the GO-DSS system and a means of tailoring the design according to the results of the analysis. As METOC operators represent a highly trained and knowledgeable group, there is merit to conducting a comparative study to compare both inexperienced and highly experienced METOC operators, to determine whether there are significant differences in the ways in which the operators approach and use the information presented in the system. This type of comparative analysis would yield critical information as to the specific elements utilized during operator interactions and further guide the design and development of the GO-DSS system. Comparative studies have yielded critical information regarding differences in approaches and decision making strategies among sonar operators [27]. This joint study, funded by the US Office of Naval Research, focused on examining the differences between US and UK Navy novice and expert sonar operators. This study demonstrated that there are cultural and experiential influences that shape a sonar operator’s performance and the ways in which they approach a signal classification task and how they interact with a display system. Thus, it would be informative to analyse the ways METOC forecasters interact and integrate information obtained during their interactions with the data, models and interface display; examine the
- 27 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
most effective layout for the display, and how operators compare charts / forecasts during their task performance. Visual displays play a pivotal role in METOC future forecasting and decision making as they seek to build forecasts based on historic and real time current data. Comparison among models is a critical means of evaluating meteorological and oceanographic information over time. Thus, the ways in which information is displayed and the number of models that can be compared at any given time would allow the operator to evaluate forecasts in a synchronous manner. Further, the ability to display several models at the same time would afford the operator the means to identify potential areas of conflict which, if highlighted, could provide the operator with an intuitive guide to query and explore potentially contradictory information. There is a need to focus on evaluating the visual presentation of information in an encoded icon pattern to determine whether it facilitates operational understanding and performance that enables a novice to behave in an expert-like manner. Events occur in a complex environment as a dynamic constellation of events versus a series of discrete events. The human is adaptive to changes but may often be overwhelmed by information and/or competing tasks. There is a limit to the amount and type of information that can be processed in a limited period of time, an idea originally introduced by Miller [23] and later elaborated on by Cowan [24] and Baddeley [25]. Today’s METOC decision maker is often left to their own devices and deciphering capabilities to sift through tons of processed information to develop a cohesive, and occasionally coherent understanding of the situation. Therefore, there is a need to develop a system design that can support the decision maker who must function with a plethora of data, under conditions of uncertainty and with restricted resources at their disposal. The designer must identify critical features and factors that can contribute to the development of a system that will facilitate situational awareness, decision making and training. The emerging field of visual analytics uses interactive computer interfaces to optimize analytical reasoning processes for strategic planning [28][29]. This union of cognitive psychology and computer science yields intuitive data representations for decision support tools that enable a decision maker to visualize synthesized information from massive, dynamic, ambiguous, and often conflicting data. These data representations allow the decision maker to create and test competing hypotheses, enabling them to detect the expected and/or unexpected; provide timely, defensible assessments and communicate such assessments effectively for action. The goal of any decision support tool is to enhance the operator’s decision making by reducing the time allotted to process information, increase the level of accuracy and optimize decision making. The long term goal is to design automated systems that will embed expert decision making strategies such that these systems will facilitate and accelerate training, performance and knowledge management in the distributed operational environment. There are significant advances in information visualization tools such as those provided in “Apps” and “Widgets” for smart phones which could be integrated in the display design itself to provide real time geospatial information and facilitate interactions among
- 28 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
operators. The utilization of such “Apps” and “Widgets” would facilitate the ability of the METOC operator to forecast weather and visualize assets in a synchronous manner. This would provide the METOC operator with a capability to monitor oceanographic data (e.g., wind velocity, wave height, etc.), as well as monitor traffic density and assets (manned and unmanned) moving within the region of interest (e.g., ports, harbours, areas of interest) in real time. Imagine a METOC “smart phone” display where you can integrate information from disparate domains in one display. The ability to visualize the environment, maintain tactical awareness of assets and an understanding of the traffic environment are critical to the METOC operator’s ability to gain an understanding of the complex operational environment. This GO-DSS system will provide an intuitive, visual means of capturing changes in thresholds of environmental and oceanographic activity, anomalies in the environment (e.g., changes in wind velocity, wave height, and locations of key assets of interest, etc.). The presentation could be encoded as visual icons, bars which are presented to show an increase in environmental activity in the region. Thus, it would be helpful to the operator to provide a means of evaluating the “accuracy/goodness” of the information. The system should provide the operator with a level of confidence and risk metric encoded in the display and a means for the operator to query the system. Such a feature embedded in the visual display would enable the operator to select and evaluate each proposed model and assess the level of risk associated with each selection. The level of uncertainty would be represented as an icon (e.g., colour coded to reflect areas of high level of uncertainty = Red; medium levels of uncertainty =Yellow; lowest level of uncertainty=Green) that could be selected and examined to reveal the level of risk and confidence for each model. This would provide decision support as the operator investigates each alternative option. The GO-DSS system will expedite their understanding of the operational environment in ways that heretofore were unavailable due to the complexity and disparity of the data collected in the environment. The GO-DSS tool will exploit information from the environment and provide tipping points to the operator/decision maker, as well as provide an ability to test hypotheses, query the system and inform more accurate decision making abilities. We contend that the results of this study could provide significant insight into the development of a dynamic decision making tool for METOC operators that could also serve as a training tool. The NICOP grant afforded the opportunity to collaborate on the evaluation of the GO-DSS system and the development of the research protocol and documentation to conduct the research. However, given the limitations and constraints of research guidelines, time and funding levels, additional research is merited. We contend that the results of this research could provide significant improvement to current METOC tactical decision aids and further facilitate the development of training methods for METOC operators who must perform their tasks across unknown terrains and regions where they lack experience and training in advance of their assignments. We intend to pursue this study as we seek additional funding to support this research effort. The questionnaires and analysis tools are already developed for data collection and
- 29 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
will be used for future NATO CMRE at-sea experiments as opportunities emerge. As a result of this NICOP study, we have the foundation to collaborate on the development of a white paper to submit to ONR in support of funding the study of the GO-DSS system. We contend that the results of these studies will inform the development of an advanced GO-DSS system that would have significant implications for future military operations both for NATO, US Navy and multinational military operations from mission planning to mission execution.
- 30 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Acknowledgement This work relates to Department of the Navy Grant N62909-12-1-7083 issued by Office of Naval Research Global. The United States Government has a royalty-free license throughout the world in all copyrightable material contained herein.
- 31 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
References [1] Grasso, R., Cococcioni, M., Mourre, B., Chiggiato, J., Rixen, M., (2011). A Maritime Decision Support System to Assess Risk in the Presence of Environmental Uncertainties: the REP10 Experiment, Ocean Dynamics, doi: 10.1007/s10236-011-05126. [2] Julier, S.J., Uhlmann, J.K. (2004) Unscented filtering and nonlinear estimation. Proc. IEEE 92(3):401–422. doi:10.1109/, JPROC.2003.823141 [3] Grasso, R., Giannecchini, S. (2006) Geo-spatial Tactical Decision Aid systems: fuzzy logic for supporting decision making. Proceedings of the 9th IEEE International Conference on Information Fusion, 10–13 July, Florence, Italy. pp. 1–8. doi:10.1109/ICIF.2006.301754. [4] Duda, R.O., Hart, P.E., Stork, D.G. (2000) Pattern classification, 2nd edn. Wiley, New York. ISBN 978-0-471-05669-0. [5] Cordon, O., del Jesus, M.J., Herrera, F. (1999) A proposal on reasoning methods in fuzzy rule-based classification systems. Int. J. Approx. Reason. 20:21–45. doi:10.1016/S0888-613X(00)88942-2. [6] Mamdani. E.H., Assilian, S. (1975) An experiment in linguistic synthesis with a fuzzy logic controller. Int. J. Man–Machine Stud. 7(1):1–13. doi:10.1016/S00207373(75)80002-2. [7] Grasso, R, Cococcioni, M, Rixen, M, Baldacci, A (2011). A Decision Support Architecture for Maritime Operations Exploiting Multiple METOC Centres and Uncertainty, International Journal of Strategic Decision Science, vol. 2, no. 1, pp. 1-27, url:http://dx.doi.org/10.4018/jsds.2011010101. [8] Endsley, M.R. (1995) Toward a theory of situation awareness in dynamic systems. Human Factors 37(1):32–64. doi :10.1518/001872095779049543. [9] Norman, D.A. "Cognitive Engineering," in D.A. Norman and S.W. Draper, eds., User Centered System Design: New Perspectives on Human-Computer Interaction, Lawrence Erlbaum Associates, Hillsdale, N.J., 31-61, 1986. [10] Smallman, H., St. John, M. (2005). Naïve Realism: Misplaced Faith in realistic Displays. Ergonomics in Design, vol. 13, no. 3, pp. 6-13. [11] Vicente, K.J. (1990) . A few implications of an ecological approach to human factors. Human Factors Society Bulletin, 33 (1).
- 32 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
[12] Vicente, K.J., Rasmussen, J. (1990). The Ecology of Human-Machine Systems II: Mediating “Direct Perception” in Complex Work Domains. In: Ecological Psychology, Lawrence Erlbaum Associates, Inc., pp. 207-249. [13] Wickens, C.D., Carswell. C.M. (1995). The proximity compatibility principle: Its psychological foundation and relevance to display design. Human Factors, 37, 473494. [14] Militello, L.G., Hutton, R.J.B. Applied Cognitive Task Analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Ergonomics Special Issue: Task Analysis,vol 41, # 11, 1618-1641, 1998. [15] Hayter, A.J. (1984) A proof of the conjecture that the Tukey–Kramer multiple comparisons procedure is conservative. Ann Statist. 12(1):61–75. [16] Benjamini, Y., Braun, H. (2002) John W. Tukey's contributions to multiple comparisons. Ann. Statist. 30(6):1576–1594. doi:10.1214/aos/1043351247. [17] Davis, M., Risley, C. (2006). Information Visualization:The State of the Art for Maritime Domain Awareness. Defence R&D – Halifax, CA. Report. MDA Document Number RX-RP-52-4186. [18] Klein, G.A., Calderwood, R., MacGregor, D. Critical Decision Method for Eliciting, Knowledge," IEEE Systems, Man, and Cybernetics, SMC-19, 462-472 (1989). [19] Klein, G.A. Sources of power: How people make decisions. Cambridge, MA: MIT Press, 1998. [20] Maciejewski, R., Rudolph, S., Hafen, R., Abusaiah, A., Yahout, M., Ouzzani,M., Cleveland, W.S, Grannis, S.J., Ebert, D.S. (2010). A visual analytic approach to understanding spatiotemporal hotspots. IEEE Trans. Visualization and Computer Graphics, Vol.16, Issue 2, pp. 205-220. [21] Roth, E. M., Patterson, E.S. & Mumaw, R. J. Cognitive Engineering: Issues in User-Centered System Design. In J. J. Marciniak (Ed.), Encyclopedia of Software Engineering, 2nd Edition. New York: Wiley-Interscience, John Wiley & Sons. [22] Simon, H. A. (1996). The Sciences of the Artificial (2nd Ed.). Cambridge, MA: MIT Press. [23] Miller, G.A. (1956). The magical number seven plus or minus two: some limits on our capacity to process information. The Psychological Review, 63: 81-97. [24] Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185 [25] Baddeley, A.D., Hitch, G.J. (1974). Working Memory, In G.A. Bower (Ed.), The Psychology of learning and motivation: advances in research and theory (Vol. 8, pp. 4789), New York: Academic Press. - 33 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
[26] Green, D.M., Swets, J.A. (1966). Signal Detection theory and Psychophysics. New York: Wiley. [27] Masakowski, Y.R. & Hardinge, N. (2000). Comparing the Cognitive DecisionMaking Strategies of US and UK Naval Sonar Operators. International Applied Military Psychology Symposium (IAMPS) Symposium Proceedings, Croatia. [28] Masakowski, Y.R. & Kaschub, C. E. (2008). The Role of Perceptually Salient Features Embedded in Visual Displays: Implications for System Design & Decision Making. [29] Cook, Thomas J. J., & Cook K. A. (2005). Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Computer Society Press, Los Alamitos, CA.
- 34 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Annex A NICOP amendments In preparation for conducting the NICOP research project, Dr. Masakowski submitted a formal research protocol to NUWC’s Institutional Review Board (IRB). (Appendix A) The NUWC IRB chair, following consultation with ONR IRB, informed Dr. Masakowski that approval from the Italian Navy would also be required per US DoD requirements as stated in SECNAV Instruction 3900.39D. SECNAV Instruction 3900.39D, paragraph 6.i., provides direction regarding requirements for research involving human subjects who are not U.S. citizens or DoD personnel, conducted outside the United States, and its territories and possessions, require permission of the host country and an ethics review by the host country, or local Naval IRB with host country representation. Upon arrival at CMRE, Dr. Masakowski met with Dr. J. Miller, Head of Applied Research Department at CMRE, to discuss this issue. Dr. Miller and Dr. Masakowski subsequently learned that the Italian Navy does not have an equivalent IRB approval process. ONR Global was informed and Dr. Livingston, ONRG NICOP sponsor made the decision not to pursue this request with the Italian Navy in order to avoid initiating any potential conflict between CMRE and the Italian Navy that might emerge due to this request. Dr. Masakowski submitted an amendment to Dr. E. Livingston who approved the amendment to waive Task 5 of the proposal without penalty.
- 35 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Annex B FR-TDA questionnaire 1 Please complete the following questionnaire to provide feedback regarding the effectiveness of the FR-TDA system. Risk Assessment Display: Does the visualization display support decision making 1 1
2
3 4
5
6
Agree 2
Neutral 3
4
5
Disagree 6
N/A 7
Display colours were helpful to evaluate the environment and risks conveyed Display was too cluttered to understand Display was too complex Display was intuitive and easy to use; helped me understand the environment more quickly Display supported my decision making Display failed to support my decision making
Comments: ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________
- 36 -
NATO UNCLASSIFIED
CMRE-FR-2012-009
NATO UNCLASSIFIED
Annex C FR-TDA questionnaire 2 FR-TDA system display questionnaire Summary: please provide your views on the displays, whether positive or negative, and provide an example:
________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________
Comments & Recommendations: please provide your comments and recommendations: ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________
- 37 -
NATO UNCLASSIFIED
Document Data Sheet Security Classification
Project No. NATO UNCLASSIFIED
Document Serial No.
EKOE
Date of Issue
CMRE-FR-2012-009
Total Pages December 2012
43 pp.
Author(s)
Masakowski, Y.R., Grasso, RT. Title Knowledge elicitation for fuzzy rule based decision support systems and system interface evaluation and design.
Abstract This paper summarizes the process for the design and development of a fuzzy rule-based tactical decision aid (FR-TDA) system which is part of the CMRE’s GO-DSS framework and the design of an enhanced interface to support decision making and maritime operation planning. The FR-TDA system under study will provide the user with geospatial meteorological and oceanographic (METOC) impact surfaces as well as risk and decision confidence indexes to support a wide spectrum of maritime operations. In this work, underwater glider deployment and recovery operations are considered as a case study. There are a number of issues involved in selecting the optimum path for glider deployment and recovery. Among these issues, it is critical to attain an understanding of the meteorological and oceanographic environment in which the gliders are operating. This study is aimed at developing a decision support tool that will assist the METOC operator with planning an optimal course of action in environmental operations by means of evaluating the environmental picture to optimize glider performance. There are two significant Human Factors (HF) issues related to developing such a decision support system (DSS): 1) de-conflicting uncertainty and 2) reduce time to decision making while increasing accuracy. In order to address the first issue, it is essential to identify the critical factors/cues used by experienced METOC Subject Matter Experts (SMEs) during their evaluation of METOC environmental data embedded with noise, clutter and uncertainty. Knowledge elicitation of SMEs for system rule bases from domain experts will facilitate mapping information usage of METOC environmental information into the DSS. The design of the system interface will require information from SMEs regarding the level of uncertainty and operational constraints and the level of significance of individual variables which they integrate during their decision making processes. The design of geographic information system (GIS) interfaces and assessment of their effectiveness is aimed at providing both an increase in system and human performance. That is, the ultimate goal is to design a DSS that optimizes the path for the glider, as well as optimizes the human’s decision making processes. This study will explore the METOC expert’s approach to mission planning for glider operations and their ability to assess and predict the environmental impact on maritime operations and provide an optimal course of action by means of METOC forecast models, as well as facilitate the development of an intuitive interface, aimed at integrating time as a critical variable for decision making. Keywords Decision support, fuzzy logic, uncertainty, interface design
Issuing Organization Science and Technology Organization Centre for Maritime Research and Experimentation
Tel: +39 0187 527 361 Fax:+39 0187 527 700
Viale San Bartolomeo 400, 19126 La Spezia, Italy
E-mail:
[email protected]
[From N. America: STO CMRE Unit 31318, Box 19, APO AE 09613-1318]
