presenting sensitivity in the form of graphs, charts, or surfaces. [3]. Generally, they are .... Figure 5: a radar graph show the influence five input variables namely ...
Interactive Visualization Techniques for Exploring Model Sensitivity Mohammad Daradkeh
Clare Churcher
Alan McKinnon
PO Box 84 Lincoln University Lincoln 7647 Canterbury
{daradkem, churcher, mckinnon}@lincoln.ac.nz
ABSTRACT Information visualization can play an important role for exploring model sensitivity to facilitate decision making under uncertainty. It provides a means for graphically exploring the relationships between the output and the inputs of a model and to determine how “sensitive” a model is to changes in the values of the input. The literature shows how sensitivity analysis can contribute to improved decision making, but little can be found about the advantages of exploring model sensitivity visually to aid the decision maker. Ultimately, the goal of this work is to develop effective interactive visualization techniques to assist people who are using models for decision making but who need to explore the often complex relationships between the values of model variables and the model output.
Categories and Subject Descriptors I.6.9 [Simulation, Modeling, and Visualization]: Visualization - information visualization, multivariate visualization, visualization techniques and methodologies. H.5.2 [Information Interfaces and Representation (HCI)]: User Interface – graphical user interfaces, interaction styles.
Keywords Information analysis.
visualization,
Interaction
design,
Sensitivity
1. INTRODUCTION Models come in many varieties and forms, ranging from the simple to the complex. Whatever category they are in, all models share the distinction of being simplifications of more complex realities that should, with proper use, result in a useful decision-making aids. Different types of models, characterized by types of input and output variables and the relationships between them, have been developed. These include static, dynamic, continuous, discrete, linear, nonlinear, deterministic, and stochastic models. All of these types are applied extensively in the context of decision making. For example, financial managers use net present value and internal rate of return in
analysing investment alternatives. Operations managers have precedence diagrams, decision trees, and a host of other models they can use to make better decisions. By their nature, decisions involve uncertainties arising from different sources. The input variables used to describe a model, on which a decision may be based, are often not 100% accurate. The uncertainty may affect the output of the model as well as the decisions based upon that output. In addition the model itself will almost always be an approximation of the real problem. Understanding how these uncertainties affect uncertainty in the model output is important for building confidence in the output of the model and the resulting decisions. The effect of uncertainty in the parameters of a model can be investigated by subjecting the model to a sensitivity analysis. The robustness of a decision that is based upon a model is not dependent on the effect of varying just a single parameter, but also on the effect of varying all parameters within a given set of values. Many models also include complex interactions so that the sensitivity due to some variables may depend on the values of others. Therefore, to assess the sensitivity of a model's output to variation in one or more inputs the decision maker needs to not only be able to determine the affect of individual parameters on the output, but also explore how that effect depends on the value of other parameters. Although decision-makers will have relevant general knowledge, they may not be domain experts so sensitivity results need to be expressed, in a clear and informative way. The most important output of sensitivity analysis is insight. The presentation and interpretation of sensitivity analysis results are important for helping an analyst or a decision-maker better understand the results. Such insight can help decision makers weigh choices in a more informed and effective way. Simple and understandable presentation formats for communicating sensitivity analysis results would be helpful. We aim to develop effective interactive visualization techniques to help decision makers explore model sensitivity to simultaneous variations of input variables. We will take into account the interaction between input variables as well as the impact of different sources of uncertainty on the overall uncertainties in model output.
2. SENSITIVITY ANALYSIS This paper was published in the proceedings of the New Zealand Computer Science Research Student Conference 2008. Copyright is held by the author/owner(s).
NZCSRSC 2008, April 2008, Christchurch, New Zealand.
Sensitivity analysis can be defined as the study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, among model inputs [1]. Similarly, it can be defined as the assessment of the impact of changes in input values on model outputs [2]. If a small change in the value of an input variable result in a relatively large change in the output, the output is said to be sensitive to that
10
M. Daradkeh, C. Churcher, and A. McKinnon
variable. Sensitivity analysis methods have been applied in different disciplines, including complex engineering systems, economics, physics, social sciences, medical decision making, and others [2, 3]. Sensitivity analysis can be performed prior to model development, during model development, and when the model is applied to a specific problem [4]. Sensitivity analysis is performed for various reasons. It can be used to increase the confidence in the model and its predictions, by providing an understanding of how a change in a variable causes a change in the dynamic behavior of the model over different sets of variable values [4]. In addition, it can play an important role in model verification and validation throughout the course of model development and refinement [5, 6]. Also, it provides insight into the robustness of model results when making decisions [3]. As an aid in decision making, sensitivity analysis can help in testing the robustness of a supposedly optimal solution by: identifying critical values, thresholds or break-even values where the optimal strategy changes, identifying the most sensitive or important decision variables, investigating suboptimal solutions, developing recommendations which are more flexible, comparing the values of simple and complex decision strategies, and assessing the risk of choosing a particular strategy or scenario [7].
analysis [3]. A brief discussion about these different classifications is given here.
2.1.1 Screening Methods Screening methods are usually performed to make a preliminary identification of the most sensitive model inputs, in order to come up with a shortlist of influential inputs [8]. They are usually applied as a first step when the model is expensive to compute and contains a large number of input variables [2, 4, and 8]. Often, only a few of the input variables have a significant effect on the model output. Screening methods can be used to identify approximately, and with low computational effort, a subset of input factors that is most likely to have a strong effect on the model output [9]. However, such methods provide only qualitative sensitivity measures (i.e. they rank the input variables in order of importance, but do not quantify how much a given variable is more important than another) [10]. Even though, they are economical from a computation point of view, they are often relatively simple and may not be robust to key model characteristics such as nonlinearity, thresholds, and interactions [8].
Questions the practitioner should ask when planning a sensitivity analysis include [3, 4]:
Several screening methods to analyse sensitivity have been proposed in the literature [11]. Typical screening analysis methods include local mathematical methods such as DSA, NRSA, and the Morris method [12] that are readily available in commercial software packages such as Microsoft Excel add-ins Crystal BallTM and SIMLAB [13] . To give an idea about how screening methods work, a brief description of Morris method is given here. This method varies one factor at a time over the space of the variables. For each variation ∆xi, an estimate of the effect on the output is computed (∆y). The average µ of all ∆y for a given variable xi is then computed to yield a global effect of y. By computing the standard deviation of the same set of ∆y one obtains an estimate of non linear and interaction effects. The method requires a total number of model evaluations that is of the order of k, O(k), where k is the number of model inputs.
When should sensitivity analysis be performed?
2.1.2 Local Methods
How should a model be prepared to facilitate sensitivity analysis?
Local sensitivity analysis focuses on relatively small perturbations near a fixed point in the model domain called a nominal point [2, 8, 9]. This point should be chosen very carefully taking into account the goal of the sensitivity analysis [2]. Local sensitivity analysis looks at the local impact of each input variable on the model output. This is usually carried out by individually varying only one of the model inputs at a time over a small interval around a nominal value, while keeping all other inputs at their nominal or base-case values [3]. The local sensitivity approach is applicable mainly to linear models [1]. However, a non-linear model may exhibit linear behaviour, when the perturbations are small so in that case the approach may be valid.
In many cases, the variability or uncertainty in model output is influenced significantly by only a subset of the model inputs that are subject to variability or uncertainty. Sensitivity analysis of model input variables can serve as a guide to identifying the important uncertainties that are often associated with variables in models for the purpose of determining the direction of future data collection activities or research. In addition, sensitivity analysis can provide insight regarding the key contributors to variability in the model output [3].
What are appropriate sensitivity analysis methods? How should the results of sensitivity analysis be presented and interpreted? It is beyond the scope of this paper to answer all of these questions. However, we aim to develop new approaches to sensitivity analysis visualization, aimed primarily at decisionmakers who are basing their decisions, at least in part, on a model with which they have been provided.
2.1 Classification of Sensitivity Analysis Methods Sensitivity analysis methods can be classified in different ways. Following [1] they are broadly classified with respect to the application and scope as screening, local, and global methods. Alternatively, methods can be classified based on the methodology of sensitivity analysis technique as mathematical, statistical (or probabilistic), and graphical [2]. Each sensitivity analysis method has its advantages and disadvantages. Selection of a sensitivity analysis method depends on model characteristics, the computational time needed to evaluate the model, available resources, and the objectives of the sensitivity
Nominal range sensitivity analysis method (NRSA) [2] is a well known technique in this category. This technique is applicable to deterministic models to evaluate the effect on model outputs of varying only one of the model inputs across its entire range of plausible values, while holding all other inputs at their nominal or base-case values [3]. The sensitivity can be represented as a positive or negative percentage change compared to the nominal solution. The sensitivity can be repeated for any number of individual model inputs. The sensitivity index is calculated as follows:
Interactive Visualization Techniques for Exploring Model Sensitivity Output max input − Output min input Sensitivity =
(1) Output
nominal input
2.1.3 Global Methods Global sensitivity analysis methods allocate the output variability to the variability of the input taking into account all the variations ranges of the inputs and interactions among them [1, 2, and 3]. Global sensitivity analysis methods must have the following two properties: (1) the sensitivity is measured over the entire range of each input variable, and (2) all the input variables are varied at the same time [1]. Sobol’s methods [14, 15] can be classified as a global sensitivity analysis technique. They are also variance-based methods based upon “Total Sensitivity Indices” (TSI) that take into account interaction effects. The TSI of an input is defined as the sum of all the sensitivity indices involving that input. The TSI includes both the main effects as well as interaction effects. For example, if there are three inputs A, B, and C, the TSI of input A is given by S(A) + S(AB) + S(ABC), where S(x) is the sensitivity index of x. S(A) refers to the main effect of A. S(AB) refers to the interaction effect between A and B. S(ABC) refers to the interaction effect between A, B, and C [3]. The general procedure for applying Sobol’s methods can be described as follow: Given a model in the form of y=f(x1x2,…,xk), where the inputs xi’s are uncorrelated, y can be defined as the realization of a probabilistic process obtained by sampling each of the xi from its respective probability distributions. Sobol’s method defines sensitivity indices based on the decomposition of the output variance into terms due to either single input effect or joint effects of more than one input [15].
2.1.4 Mathematical methods Mathematical methods for sensitivity analysis typically address the local or linear sensitivity of the output to perturbations or ranges of individually varied inputs [3]. They assess the sensitivity by calculating the output for a few values that represent the possible range of the inputs [16]. These methods do not address the variance in the output due to the variance in the inputs. They are useful in screening the most important inputs and to identify inputs that require further data acquisition or research [17]. Different mathematical techniques can be found in the literature such as nominal range sensitivity analysis (NRSA) and differential sensitivity analysis (DSA) [2]. In DSA the local sensitivity is calculated at one or more points in the parameter space of an input keeping other inputs fixed. DSA is performed with respect to some point x in the domain of the model. A small variation ∆x with respect to the point value of a model input, such as a change of plus or minus one percent, can be used to evaluate the corresponding change in the model output. Thus, the sensitivity index may be calculated as: Output x+ ∆x − Output x- ∆x Sensitivity =
(2) Output x
2.1.5 Statistical methods Statistical methods for sensitivity analysis address the effect of variance in inputs on the output variance. Analyses of this type involve running simulations using probability distributions
11
assigned to the inputs, and then assessing the impact of variance in inputs on the output distribution [18]. Depending upon the method, one or more inputs are varied at a time. Statistical methods allow one to identify the effect of simultaneous interactions among multiple inputs. These methods are useful in identifying the effect of simultaneous interaction among multiple inputs. A variety of sampling techniques, such as Monte Carlo simulation and Latin Hypercube sampling [19], can be used to generate distributions for model inputs.. A well-known technique is the Fourier Amplitude Sensitivity Test (FAST) [4]. The main idea behind the FAST method is to use the properties of Fourier series to approximate the variance in output values and apportion the output variance to variance of the model inputs. Application of FAST involves defining a set of transformation functions and angular frequencies for model inputs. FAST uses the defined transformation function of each input for sampling during a probabilistic simulation of a model. Some statistical software packages are available for application of FAST to a model. For example, SIMLAB has the ability to perform FAST [13].
3. Sensitivity Analysis Visualization Graphical methods for sensitivity analysis provide a means for presenting sensitivity in the form of graphs, charts, or surfaces [3]. Generally, they are used to visually display the relationships between the output and input variables [17]. Graphical methods for sensitivity analysis can be used with any kind of model to represent complex dependencies between inputs and outputs. In addition, they can be used as a screening method to guide the selection of appropriate sensitivity analysis methods. They can also complement mathematical and statistical methods [3]. Visualization of model sensitivity can be thought of as a multidimensional problem for which there are many possible techniques. For model sensitivity we usually consider only one output from a model. Despite the vast amount of multidimensional visualization techniques available, little can be found for exploring model sensitivity to the variability and uncertainty of the input variables. In this section, we will present some of the known approaches where visualization is used for sensitivity analysis.
3.1 Tornado Diagram The purpose of a tornado diagram is to graphically show which variables have the most influence on model output and rank them in order of importance [19]. It consists of stacked horizontal bars, each one representing the sensitivity of one independent variable. The left and right bar ends indicate the corresponding upper and lower bounds of the model output corresponding to the upper and lower bounds of the input variable. The length of the bar indicates the variable's total effect on the model output. The bars are arranged vertically according to decreasing sensitivity. The model output has a base value which is calculated for the base values of all the input variables and displayed as a vertical line on the diagram. By knowing which inputs are most sensitive, the user is better informed and can decide where to invest effort in reducing uncertainties. However, the underlying assumption of the tornado diagram is that each of the input variables is independent. If the variables are not independent, a set of dependent variables may misleadingly appear lower on the diagram than they should. Therefore, Tornado diagrams are
12
M. Daradkeh, C. Churcher, and A. McKinnon
mainly used for graphically displaying the results local sensitivity analysis techniques. A typical diagram is shown in Figure 1[HREF1]. It shows the sensitivity of Net Present Value (NPV) to the variation in the Real Rent Escalation rate holding all other rates constant. The bars show that the NPV is very sensitive to changes in the Real Rent Escalation rate but changes in Refinance points’ rate have little effect on NPV.
plots arranged in rows and columns. Figure 3 [19] is an example of scatter plots showing the influence of five input variables, namely, battery(bat), lamp bulb(bulb), starter motor(strtr), car's ignition(ignitn), and headlight(headlite) on the model's output variable, engine head(ign-head), as ranked correlations.
Figure 3: a scatter plot matrix show the relationship between ign-head as an output and, respectively, bat, bulb, strtr, ignitn and headlite as input variables [19].
Figure 1: Tornado diagram showing the sensitivity of NPV to the variation of Real Rent Escalation rate [HREF1].
3.2 Scatter Plots
One limitation of the scatter plot matrix is that it cannot show interaction effects with another variable. This is the strength of the conditioning plot. A conditioning plot, also known as a coplot or subset plot, is a plot of two variables conditional on the value of a third variable (called the conditioning variable). It is useful for displaying scatter plots for groups in the data. Although these groups can also be plotted on a single plot with different plot symbols, it can often be visually easier to distinguish the groups using the conditioning plot. The conditioning variable may be either a variable that takes on only a few discrete values or a continuous variable that is divided into a limited number of subsets. Figure 4 [HREF3] shows a conditional scatter plot of torque versus time conditional on the value of temperature. From figure 4, we can see that the sensitivity of torque to the variation in time depends on the value of temperature. As the temperature value increased, the sensitivity of torque increased.
Scatter plots are used for visual assessment of the influence of individual inputs on an output [3]. They visualize the relationship between a single input variable and an output by plotting points with the input value on the x-axis and the output value on the y-axis [9]. They can show various kinds of relationships, including positive (rising), negative (falling), and no relationship. The closer the correlation is to 1 or -1, the stronger the correlation, or the stronger the relationship between the input and output. Figure 2 [HREF2], shows the relationship between var_x as input variable and var_y as output variable.
Figure 4: a conditioning plot of torque versus time conditional on the value of temperature [HREF3]. An advantage of conditioning plots is that they may reveal potentially complex dependencies. A potential disadvantage is that they may be tedious to generate and inspect if there are a large number of inputs. Some software packages will automatically generate multiple scatter plots and provide interaction techniques to enable the user to select the input variables to be graphed [3]. Figure 2: Scatter Plot shows the relationship between var_x vs. var_y [HREF2]. A single scatter plot usually has only two axes and can therefore encode only two variables in space. To display more than two variables at a time, several scatter plots can be combined to form a scatter plot matrix. It consists of a number of scatter
3.3 Radar Graphs Radar graphs provide another way of showing the influence of independent input variables on model output [19]. A radar graph, also known as a star or spider graph, is laid out in a circular fashion, rather than the more common linear
Interactive Visualization Techniques for Exploring Model Sensitivity arrangement. Figure 5 [19] is a radar graph showing the dependence of the model output variable, ign-head, on five input variables, bat, bulb, strtr, ignitn and headlite. The axis lines start in the centre of a circle and extend to its border. Each input variable corresponds to an axis in the graph. The input variable with the highest positive influence on the model output (e.g. highest positive correlation) is plotted furthest from the center of the graph, and the variable with the highest negative influence (e.g. highest negative correlation) is plotted closest to the center of the graph. The input variable that has small effect on the output variation (e.g. lowest correlation) is plotted in the middle of the axis line. Interpreting Figure 5 we have that the input bat, which is plotted near to the middle of the axis line, has the least effect on the ign-head, where as, strtr, with the longest line, has the largest positive effect. The very shortest lines have the greatest negative effect. With the variables with the largest effects being either the longest or shortest lines is difficult to see at a glance the variables that have a small effect on the output.
13
Figure 6: Conditional parallel coordinates shows the influence of bat, bulb, strtr, ignitn and headlite on ign-head [19].
4. PROPOSED WORK We propose to focus on the development of new visualization techniques to allow the exploration of model sensitivity in a clear and informative way. We want the techniques to be easily understood and applied by people who are using models for decision making. The visualizations will include the contribution of different sources of uncertainty in model input variables to the overall uncertainty in model outputs, and the influence of varying input variables on the output variability taking into account the presence of variable interactions. In this section, we will describe an initial prototype designed for visualizing model sensitivity. Figure 5: a radar graph show the influence five input variables namely, bat, bulb, strtr, ignitn and headlite on the output variable, ign-head, as rank correlation [19]. Radar graphs are helpful for small-to-moderate-sized multivariate data sets. Their primary weakness is that, when the number of variables and dimensions increase they tend to be overwhelming.
3.4 Parallel Coordinates A parallel coordinates plot allows the presentation of all results in a single plot frame and also facilitates the recognition of interactions between variables [17]. It provides a representation for a multidimensional distribution in a two-dimensional plot [21]. In a parallel coordinates plot the axes corresponding to the input variables and the output variable are drawn vertically. We can infer the sensitivity of the output variable by analyzing the distribution of the points on the vertical axes. The real strength of parallel coordinates plots is that they support interactive conditionalization; that is, the user can define regions on the various axes and select only those samples which intersect the chosen region. For example, Figure 6 shows the sensitivity of ign-head value to the variation in bat, bulb, strtr, ignitn, and headlite after conditionalization on high values of ign-head. It can be seen that a large range of headlite values leads to similar values of ign-head. In contrast, small variation in ignitn leads to small variation in ign-head. This is means that ign-head is more sensitive to the variation in ignitn than to the variation in headlite.
The basic idea of our proposed work is to use animation techniques where a visual display like a chart or graph is systematically varied showing results of changing model variables. We suggest using a set of horizontal bars to display the model sensitivity to the variation in input variables, one bar for each input variable. Each bar illustrates the relative sensitivity of one input variable. A vertical line is drawn representing the value of the model output at each time when we vary the model’s input variables. Since input variables have different scaling and different uncertainty, we suggest varying each input variable by a fixed percentage. This is similar to tornado diagram but we will allow the user to change the base values interactively and we will take into account the interaction between input variables. Figure 8 shows a possible prototype applied, as an example, to exploring the sensitivity of a financial model used to analyze the profitability of a project. Each bar represents the range of possible profits generated by varying the input variables by 10% and the position of the vertical line represents the profit at certain values of input variables. It can be clearly seen that the profit is most sensitive to changes in software cost, whereas maintenance has little effect on the profit. This can be seen by comparing the length of the horizontal bars relating to these inputs. The user can change the values of the input variables by a fixed percentage, and as a result, the position of the vertical line that represents the value of profit will be changed as depicted in Figure 9.
14
M. Daradkeh, C. Churcher, and A. McKinnon variables have on model output, and also how that effect depends on the actual values of other input variables.
Market size
This paper proposes an initial prototype of an interactive visualization tool based on an extension of the concept of a tornado diagram which will allow such exploration. It is proposed to trial this prototype on users, and based on the results, revise or expand the approach and test it further with users exploring a range of different model types.
Software cost Hardware cost
5. REFERENCES [1] Saltelli, A. 2002a, Sensitivity analysis for importance assessment, Risk Analysis, 22(3): 1-12.
Maintenance
Figure 8: The initial prototype Figure 9 shows the effect of changing the values of the input variables. The position of the vertical line has moved to represent the new value of the profit calculated from the new values of input variables and the horizontal bars lengths changed to represent the new sensitivity. From Figure 9, it can be seen that the horizontal bar related to hardware cost is now the longest. Thus, the profit is more sensitive to the variation in the hardware cost than the variation in the other input variables at these new values of the input variables.
Market size
[2] Frey, H.C. and R. Patil .2002. Identification and review of sensitivity analysis methods, Risk Analysis, 22(3):553577. [3] Frey, H.C., A. Mokhtari, and J. Zheng. 2004 Recommended Practice Regarding Selection, Application, and Interpretation of Sensitivity Analysis Methods Applied to Food Safety Process Risk Models, Prepared by North Carolina State University for U.S. Department of Agriculture, Washington, D.C. 148. [4] Ascough Ii, J.C., Green, T.R., Ma, L., Ahuja, L.R. 2005. Key criteria and selection of sensitivity analysis methods applied to natural resource models. International Congress on Modeling and Simulation Proceedings. Salt Lake City, UT, November 6-11,2005. [5] Kleijnen, J.P.C. and R.G. Sargent . 2000,. A methodology for fitting and validating metamodels in simulation, European Journal of Operational Research, 120(1):14-29.
Software cost
[6] Fraedrich, D. and A. Goldberg. 2000. A methodological framework for the validation of predictive simulations, European Journal of Operational Research, 124(1):55–62.
Hardware cost
[7] Pannell, D.J. 1997. Sensitivity analysis of normative economic models: Theoretical framework and practical strategies. Agricultural Economics 16: 139-152, at URL http://cyllene.uwa.edu.au/~dpannell/dpap971f.htm.
Maintenance Figure 9: The initial prototype after changing the values of the input variables. Interaction techniques will be used to facilitate the visualization. For example, by designing a suitable graphical user interface, the user can choose which model input variables to display, actively change the values of those variables, and change the range of the input variables. In addition, interaction techniques will be used to customize the visualization. The purpose of visualizing model sensitivity is to helping the target audience to better understand the results and gain insight into the model. The resultant diagram will be explored by different types of audiences with a range of different backgrounds and experience. Interaction techniques provide flexibility to customize the appearance of the diagram, layout, and enable users to aggregate and filter information based on various attributes.
4. CONCLUSION Despite the importance of sensitivity analysis in the development and use of models, particularly for decisionmaking, the literature shows that there has been little work done on the development of interactive visualization tools to help the decision-maker explore model sensitivity. Such tools should allow the user to explore the effect uncertainties in model input
[8] S. Wagner. 2007. Cost-Optimisation of Analytical Software Quality Assurance. PhD Dissertation, Technische Universit¨at M¨unchen, to appear. [9] S.Wagner. 2006. A Model and Sensitivity Analysis of the Quality Economics of Defect-Detection Techniques. In Proc. International Symposium on Software Testing and Analysis (IS-STA ’06):73–83. ACM Press. [10] Kioutsioukis, I., S. Tarantola, et al. 2004. "Uncertainty and global sensitivity analysis of road transport emission estimates." Atmospheric Environment 38(38): 6609-6620. [11] F. Campolongo, A. Saltelli, T. Sorensen and S. Tarantola. 2000. Hitchhiker’s guide to sensitivity analysis. In: A. Saltelli, K. Chan and E.M. Scott, Editors, Sensitivity Analysis, Wiley, Chichester (2000):15–47. [12] M.D. Morris. 1991. Factorial sampling plans for preliminary computational experiments. Technometrics 33 2 (1991):.161–174. [13] SIMLAB. 2000. Software for Uncertainty and Sensitivity Analysis, Version 1.1. Developed by the Joint Research Center of the European Commission in the Framework of the Project" Integration and Demonstration of the Uncertainty and Sensitivity Analysis software PREP/SPOP."
Interactive Visualization Techniques for Exploring Model Sensitivity [14] I.M. Sobol’. 1993. Sensitivity estimates for nonlinear mathematical models, Mathematical Modeling and Computational Experiments 1 (1993) (4):. 407–414. [15] A. Saltelli, K. Chan and M. Scott. 2000. Sensitivity analysis Probability and Statistics Series, Wiley, Chichester, UK (2000). [16] Salehi, F., S.O. Prasher, S. Amin, A. Madani, S.J. Jebelli, H.S. Ramaswamy, and C. T. Drury . 2000. “Prediction of Annual Nitrate-N Losses in Drain Outflows with Artificial Neural Networks," Transactions of the ASAE, 43(5):11371143. [17] D.A.Ababei & .Kurowicka & R.M.Cooke. 2007. Uncertainty Analysis with UNICORN .Third Brazilian conference on statistical Modelling in Insurance and Finance, Maresias, March 25-30, 2007. [18] Cullen, A.C., and H.C. Frey.1999. Probabilistic Techniques in Exposure Assessment. Plenum Press: New York. [19] R.M. Cooke and J.M. van Noortwijk. 2000. in A. Saltelli, K. Chan, and E.M. Scott.. Graphical methods , editors, Sensitivity Analysis, Chichester: John Wiley & Sons, 2000. 245-264. [20] Andersson, F.O., M. Aberg, and S.P. Jacobsson. 2000. "Algorithmic Approaches for Studies of Variable Influence, Contribution and Selection in Neural Networks," Chemo-metrics and Intelligent Laboratory Systems, 51(1): 61-72. [21] Hofer, Eduard. 1999. Sensitivity analysis in the context of uncertainty analysis for computationally intensive models. Computer Physics Communications, 117(1-2):.21-34.
6. HYPERTEXT REFERENCES [HREF1]http://www.vpistrategies.com/articles_pdf/ TrndoArt.pdf. [HREF2] http://www.statsoft.com/textbook/gloss.html [HREF3]http://www.itl.nist.gov/div898/handbook/eda/section3 /scatterc.htm
15
