Combining Disparate Information Sources when

Combining Disparate Information Sources when Quantifying Operational Security Siv Hilde Houmb Department of Computer and Information Science Norwegian University of Science and Technology Sem Sælands Vei 7-9, NO-7491 Trondheim, Norway [email protected]

Abstract Quantitative estimation of security attributes makes it possible to do cost-effective development of security critical systems. By predicting the impact and cost of potential misuses, as well as the cost and effect of security treatment strategies, one can treat security risks at the right time for the correct cost. The Aspect-Oriented Risk-Driven Development (AORDD) framework supports cost-effective development through its Bayesian Belief Network (BBN) based cost-benefit trade-off analysis and estimation repositories. Estimation of misuse and treatment strategy attributes is done using disparate information sources. The AORDD framework supports two types of information sources; empirical or observable data and expert opinions. Keywords: Quantitative security assessment, trade-off analysis, and Bayesian Belief Network (BBN).

4 provides a description of the BBN methodology. Section 5 presents the BBN based prediction system, while Section 6 presents a small example to demonstrate the approach. The paper concludes by pointing to future work as well as give a brief summary of the main contributions of the paper.

2 The AORDD Framework The AORDD framework combines risk-driven development (RDD) [16] with aspect-oriented modeling (AOM) [3]. The framework consists of the iterative development process AORDD [6], a security treatment aspect repository, an estimation repository, rules for how to annotate UML models with information used for estimation, rules for how to transfer information from the annotated UML models into the BBN topology, and a BBN- based cost-benefit trade-off analysis. In this paper we focus on the trade-off analysis, and in particular how to combine disparate information sources using BBN.

1 Introduction 3 Information sources Cost-effective development of security critical systems require quantitative estimation of operational security [11]. To estimate operational security one need to assess the cost and impact of future misuses as well as the effect and cost of different treatment strategies. The Aspect-Oriented Risk-Driven Development (AORDD) framework supports cost-effective development through its cost-benefit tradeoff analysis. The trade-off analysis is implemented as a Bayesian Belief Network (BBN) based prediction system and operational security is predicted using estimation repositories. The estimation repositories consist of data from various information sources, which are combined using three combining strategies. The paper is organized as follows. Section 2 gives a brief overview of the AORDD framework. Section 3 presents strategies for combining information sources, while Section

Since there are not sufficient amount of relevant empirical data, we need to combine information sources and reason under uncertainty. We use two main types of information sources for misuse and treatment strategy attribute estimation; empirical (“objective) or observable information and subjective expert judgments.

3.1 Empirical or observable information sources Empirical or observable data is information that represents an observation of the world or an observation done though an experiment, where the purpose is to observe the presence of a particular phenomenon. One type of observable information sources are real-time information sources, such as Intrusion Detection Systems (IDS) and honeypots

[15]. Other observable information sources are company experience repositories, public repositories, and experience from similar systems. Examples of public repositories are the quarterly reports from Senter for Informasjonssikkerhet (SIS) in Norway, CERT.com, reports from the Honeynetproject [17], and other attack trend reports. When using information from similar systems in the AORDD framework we use experts to reason about their differences, and the effect on the configuration being assessed. The result of this is documented both as annotated UML models, and a document describing the experts and their discussion of the provided opinions. This process is formalized using a documentation template.

used when combining disparate information sources. This depends on the information available. We have not established a general rule for this yet, but are looking into effects as part of a serie of experiments being conducted. Observable information sources Empirical data

3.3 Combining strategies Aggregate expert opinions and combine observable and subjective information can be done in a number of ways. In this paper we consider three combing approaches [7]. Their main difference is on whether or not observable information sources is used as input to expert judgments. Figure 1 gives an overview of the three approaches. Arrow A represent observed information given as input to expert judgments, while arrow B represent observed information used as input to combining disparate information sources. The three approaches are: 1.) Observable information sources used both as input to expert judgment and as input when combining disparate information sources (includes both arrow A and B). 2.) Observable information sources not used as input to expert judgment, but used when combining disparate information sources (not A, but B). 3.) Combining of information sources done by giving observable information sources as input to expert judgment and then aggregate expert opinions without combing it with observable data (A, but not B). The observed information given as input to expert judgments may or may not be the same information

Domain experts

Aggregating expert opinions

3.2 Subjective expert judgments Speculation, brainstorming and guessing made by experts and used as basis for structured decision processes is of relatively new date [12]. It started with the establishment of RAND Corporation in the USA after the Second World War. RAND developed two methodologies for expert eliciting, the scenario method and the Delphi method. However, later evaluations have showed that the Delphi method violates the methodological rules for common experimental science. One problematic aspect with the Delphi method is that the result from the expert judgments represents their agreed values, rather than aggregated and traceable individual estimates. When estimating misuse impact, treatment strategy effect, and cost we need to distinguish and use expert opinions from different domains.

A

B

Combining observable information sources and expert opinions

Figure 1. Strategies for combining expert opinions and observable information sources

4 Procedure sources

for

combining

information

Figure 2 gives an overview of the procedure for combining information sources. The procedure consist of three main steps; determining expert knowledge level, aggregating expert opinions, and combine information sources. An experts knowledge level is determined using the variables age, gender, years of relevant education, years of education others, years of experience from the industry, years of experience from the academia, domain A, domain B, domain C, and domain D. The set of variables chosen are based on discussions in [2, 18]. We use a weighting algorithm to specify the relations between the different variables. However, we have not yet done any experiments to verify the correctness, in terms of relevance and significance, for the choice of variables and the relation between them. In the current version all variables are given the same weight. As discussed earlier we may or may not use observable information sources as input to expert judgments. In practice this mean that experts in some cases are given relevant information when doing their estimations. Experts may choose to disregard the information. When aggregating expert opinions we use priorities, weighting, company specific priorities, and domain specific priorities to determine the utilities between the variables describing an expert. Finally, the combining of information sources is done using priorities, weighting, company specific priorities, and

domain specific priorities that describe the utilities between the different variables. In the current version the information sources supported are honeynet, prior experience, other relevant company specific experience, other relevant public open experience, public experience repositories, and subjective expert judgments.

5 The BBN methodology

Bayesian Belief Networks (BBN) have proven to be an powerful technique for reasoning under uncertainty, and have been successfully applied when assessing the safety of systems by combining information sources [4]. The BBN methodology is based on Bayes rule and was introduced in the 1980s by Pearl [14] and Lauritzen and Spiegelhalter [10]. HUGIN [8] is the leading tool supporting BBN. Bayes rule calculates conditional probabilities. Given the two variables X and Y the probability P for the variable X given the variable Y can be calculated from: P(X|Y)=P(Y|X)*P(X)/P(Y). By allowing Xi to be a complete set of mutually exclusive instances of X, Bayes formula can be extended and you can calculate the conditional probability of Xi given Y. A BBN is a connected and directed graph consisting of a set of nodes and a set of directed arcs (or links) between them. Nodes are defined as stochastic or decision variables, and multiple variables may be used to determine the state of a node. Each state of each node is expressed using probability density. The probability density expresses our confidence to the various outcomes of the set of variables connected to a node, and depends conditionally on the status of the parent nodes at the incoming edges. The nodes and associated variables can be classified into three groups: (1) Target node(s) - the node(s) about which the objective of the network is to make an assessment. An example of such a node is “Combining disparate information sources”. (2) Intermediate nodes - nodes for which one have limited information or beliefs. The associated variables are the hidden variables. Typically hidden variables represent aspects that increase or decrease the belief in the target node. (3) Observable nodes - nodes that can be directly observed or in other ways obtained. Application of the BBN method consist of three tasks; construction of the BBN topology, elicitation of probabilities to nodes and edges, and making computations. For further information on BBN see Jensen [9]. For more information on the application of BBN for software safety assessment see Gran [4].

6 Combining disparate information sources using BBN

In this section we address the construction of the BBN topology. The combining disparate information sources BBN topology consist of a top level BBN and two subnets. We start by describing the two subnets and then the top level BBN. Recall from Section 4 that the procedure for combining information sources consist of three steps; determine an experts knowledge level, aggregate expert opinions, and combine disparate information sources. Experts are described using a set of variables; gender, age, years of relevant education, years of other education, years of experience from the industry, years of experience in academia, and the domain of knowledge (in this case described by domain A, B, C, and D). Figure 3 depict the expert BBN subnet. The first part of Table 1 provides an overview of the variables and states of the nodes in the expert subnet. The set of variables used to describe an expert’s knowledge level is used when aggregating expert opinions. After each expert has given their input, the expert opinions are aggregated. We have two main types of aggregation; average (how to compute average will vary depending on values used to describe the states, e.g. for qualitative values one normally use mode) and weighted aggregation. The BBN topology supports both types of aggregation. Figure 4 depict the aggregation of expert opinions subnet. The node ”type of aggregation” determines the type of aggregation used. The nodes priorities, weighting, company specific priorities, and domain specific priorities determines the set of priorities and weighting used. The utility node (the diamond) is used to specify the utilities of the different combinations of states of the variables. The second part of Table 1 gives an overview of the variables and their associated states for aggregating expert opinions. When the expert opinions is aggregated, the aggregated values is combined with the observable information sources. Figure 5 depict the top level BBN for the combining information sources topology. The utility node specifies the utilities of the combinations of all states of all variables. The observable information sources supported are real-time data from honeynets, information from public repositories such as general incidents descriptions, company specific experience (for the exact same configuration), prior experience that is public available, and other relevant experience. The subjective information source are aggregated expert opinions. The third part of Table 1 gives an overview of the variables and their states for combining information sources.

Node age gender years of relevant education years of education others years of experience from industry years of experience academia are of expertise domain A domain B domain C domain D Node type of aggregation priorities, weighting, company priorities, and domain priorities Node honeynet, public repositories, experience company, experience public, and aggregation expert opinions

Variables under 20, 20-25, 25-30, 30-40, 40-50, over 50 male and female 1 year, 2 years, Bsc, Msc, PhD, other 1 year, 2 years, Bsc, Msc, PhD, other 1-3 years, 5 years, 5-10 years, 10-15 years, over 15 years

States 0 and 1 0 and 1 0 and 1 0 and 1 0 and 1

1-3 years, 5 years, 5-10 years, 10-15 years, over 15 years

0 and 1

database, network management, developer, designer, security management, and decision maker database, network management, developer, designer, security management, and decision maker database, network management, developer, designer, security management, and decision maker database, network management, developer, designer, security management, and decision maker database, network management, developer, designer, security management, and decision maker Variables average and weighting age, gender, years of relevant education, years of education others, years of experience from the industry, years of experience from the academia, domain A, domain B, domain C, and domain D Variables priority, domain relevance, and weighting

low, medium, and high low, medium, and high low, medium, and high low, medium, and high low, medium, and high States 0 or 1 state as given in Table 1

States 0...1

Table 1. Variables and states for the expert, the aggregation of expert opinions, and the combining disparate information sources BBN subnet

age gender years of relevant education years of education others years of experience industry years of experience academia

expert 1 expert 2 expert n-1 expert n

domain A domain B Determine expert domain C domain D knowledge level expert knowledge level

Aggregate expert opinion

priorities weighting company specific priorities domain specific priorities

aggregated expert opinion honeynet prior experience other relevant experience –CS other relevant experience –public public experience repositories

Combine disparate information sources

priorities weighting company specific priorities domain specific priorities

combined information sources

Figure 2. The inputs and outputs of the three main step of the procedure for combining information sources

Figure 3. The expert subnet

Figure 4. The aggregating expert opinions subnet

7 Combining honeypot and expert opinions when predicting number of CodeRed.F worm attack The e-Commerce platform ACTIVE [1] consist of a web server running Microsoft Internet Information Server (IIS)

4.0, an Java application server (Allaire JSP Engine), and a database server running Microsoft SQL Server.

Figure 5. Top level BBN for combining disparate information sources A common misuse for a not up to date patched IIS 4.0 is the CodeRed.F worm. The latest version of the CodeRed worm, CodeRed.F, has been spreading in the Internet since March 11th 2003 and is, in contrast to its precursors, enabled to spread forever. CodeRed.F sends its code as an HTTP request. The HTTP request exploits a known bufferoverflow vulnerability in IIS 4.0 that allows the worm to run on the target computer. Since CodeRed.F uses an old exploit to infect the index server, there exist a set of patches for the exploit. Another treatment strategy is to replace the IIS 4.0 with another webserver, or to implement a filter with rules for legal and illegal connection attempt to the web-server. In this example we use a patch that removes the vulnerability.

7.1 Information sources used We use two types of information sources in this example; real-time data from a honeypot and expert opinions from 18 domain experts. The honeypot was configured with Windows NT 4.0 operating system and IIS 4.0. The configuration of the honeypot reflects the configuration of ACTIVE. For more information on the honeypot and its configuration the reader is referred to [13]. The group of experts used where undergraduate students at Norwegian University of Science and Technology. Further information on the collection of expert judgment is provided in [7].

7.2 Elicitation of probabilities Recall from Section 5 that the BBN methodology consist of construction of the BBN topology, elicitation of probabilities to nodes and edges, and making computations. Elicitation of probabilities for the BBN is done by specifying the prior probability functions. The BBN for the combining information sources is system and domain specific, and one need to specify the prior distribution for the observable nodes, as well as determine the utilities. In the next step of the BBN methodology information is collected and inserted into the observable nodes before computations is performed. The information

inserted represent the likelihood function in the BBN methodology. The posterior probability function is obtained by computing prior and likelihood: RPposterior (Xi |Y) = (Pprior (Xi |Y)*Plikelihood (Xi |Y))/( (Pprior (Xi |Y)* Plikelihood (Xi |Y)dY). In this example we have given all combination of states the same utility, meaning that they have the same influence on the result of the computation. Figure 6 depict the utility scores for aggregating expert opinions, the utility function U1 in Figure 4.

7.3 Computation with the BBNs Computation with the BBNs is done by inserting observations in the observable nodes, and then use the rules for probability calculation backward and forward along the edges, from the observable nodes, through the intermediate nodes to the target node. Forward calculation is straight forward, while backward computation is more complicated. Backward calculation is solved using Bayes methodology (see [9] for details). We use the BBN tool HUGIN [8] to construct the BBN since manual computation on large BBN topologies is not tractable. Note that the amount of information collected before a decision is made depends on the type of decision, the resources available, the time frame, and the budget. This means that one should not spend more money on collecting information than the value of the decision. The information collected from the Honeypot relates to the frequency of CodeRed.F worms attacks and their potential impacts. The honeypot configuration detected an average of 46.67 CodeRed.F attack per hour in a 12 hour logging period. Since the situation for CodeRed.F is that the web server is either protected or not, all attack tries are successful until the server is patched. This gives CodeRed.F frequency=46.67 attacks/hour. A sample of the result of the expert judgments is given in Table 2. Computation is done by first inserting the information into the expert BBN subnet. This information is then computed and given as input to the aggregating expert opinions subnet through the node expert. In this example we use four experts. To demonstrate the approach we have only used a

Figure 6. Utility function for aggregating expert opinions Expert number 4 6 15 22

Expert description

Variable and estimation

medium level of expertise, BSc education, 0 years of experience low level of expertise, BSc education, 0 years of experience high level of expertise, BSc education, 0 years of experience low level of expertise, BSc education, 0.5 years of experience

frequency; high=0.2 frequency; high=0.0 frequency; high=0.5 frequency; high=0.4

low=0.1;

medium=0.7;

low=0.4;

medium=0.6;

low=0.2;

medium=0.3;

low=0.3;

medium=0.3;

Table 2. Estimates given using expert judgments subset of the variables to describe each expert. The experts assessed the frequency of CodeRed.F worm attack tries per hour given as the qualitative scale low, medium, and high. When combining information sources after aggregation of expert opinions qualitative and quantitative values are combined using the BBN for combining information sources, the top level BBN. In this example we use a simple aggregation approach where high level of expertise is given weight=3, medium level of expertise is given weight=2, and low level of expertise is given weight=1. Education is the same for all four experts and we therefore do not include this information in the aggregation. For demonstration purpose we have given all combination of states in the combining information sources BBN, the top level BBN, the same utility. Figure 7 depict the utility scores for combining information sources.

8 Conclusion and further work Cost-effective development of security critical systems require the use of techniques to quantify misuse effect, frequency, and potential cost, as well as treatment strategy effect and cost. Since we do not have sufficient amount of empirical information we need to combine disparate information sources. The paper have presented possible information sources, combining strategies, combining procedure, and the implementation of the procedure using BBN. The BBN topology consist of two subnets and the top level network. Experts, aggregation of experts, and combining information sources is described using a set of variables and states. An experts knowledge level is determined using a utility function that describes the relations between the variables using probability density functions (pdf). These functions are used to de-

[5]

[6]

[7]

[8] [9]

Figure 7. Utility function for combining disparate information sources

[10]

[11]

scribe potential outcomes of all combination of states in the network. The same goes for the utility functions for aggregating expert opinions and combining disparate information sources. These utilities are domain specific, meaning that some of the relations are general and can be specified based on domain experience. In other cases the utilities need to be specified for each case. The BBN topology presented in this paper is used as input to a second BBN topology in the AORDD framework, which models the problem being assessed. This BBN topology models the relations between misuse and treatment strategy. More information on this topology is given in [5]. Further work includes a series of experiments looking into the relations between the variables for the two BBN topologies, including the feasibility of the combining strategies as well as the combining procedure.

[12] [13]

[14] [15] [16]

[17]

References [1] EP-27046-ACTIVE, Final Prototype and User Manual, D4.2.2, Ver. 2.0, 2001-02-22., 2001. [2] R. M. Cooke. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, 1991. [3] G. Georg, R. France, and I. Ray. An aspect-based approach to modeling security concerns. In Workshop on Critical Systems Development with UML (CSDUML’02). Dresden, Germany, October 2002. [4] B. A. Gran. The use of Bayesian Belief Networks for combining disparate sources of information in the safety assessment

[18]

of software based systems. Doctoral of engineering thesis 2002:35, Department of Mathematical Science, Norwegian University of Science and Technology, 2002. S. H. Houmb, G. Georg, R. France, J. Bieman, and J. J¨urjens. Cost-benefit trade-off analysis using bbn for aspect-oriented risk-driven development. Accepted for the International Conference on Engineering of Complex Computer System (ICECCS2005) in Shanghai, China, 16-20 June 2005. S. H. Houmb, G. Georg, R. France, and D. Matheson. Using aspects to manage security risks in risk-driven development. In 3rd International Workshop on Critical Systems Development with UML, number TUM-I0415, pages 71–84. TUM, 2004. S. H. Houmb, O. A. Johnsen, and T. Stalhane. Combining Disparate Information Sources when Quantifying Security Risks. In Proceeding of SCI 2004, RMCI 2004, Orlando, July 2004, 2004. HUGIN: Tool made by Hugin Expert a/s, Alborg, Denmark, 2004. http://www.hugin.dk. F. Jensen. An introduction to Bayesian Network. UCL Press, University College London, 1996. S. L. Lauritzen and D. J. Spiegelhalter. Local computations with probabilities on graphical structures and their application to expert systems (with discussion). Journal of the Royal Statistical Society, Series B 50(2):157–224, 1988. B. Littlewood, S. Brocklehurst, N. Fenton, P. Mellor, S. Page, D. Wright, J. Dobson, McDermid J., and D. Gollmann. Towards operational measures of computer security. Journal of Computer Security, 2:211–229, 1993. K. Øien and P. R. Hokstad. Handbook for performing expert judgment. Technical report, SINTEF, 1998. M. E. Østvang. The honeynet project, Phase 1: Installing and tuning Honeyd using LIDS, 2003. Project assignment, Norwegian University of Science and Technology. J. Pearl. Probabilistic Reasoning in Intelligent Systems: Network forPlausible Inference. Morgan Kaufmann, 1988. L. Spitzner. Honeypot - tracking hackers. Addison-Wesley, 2003. K. Stølen, F. den Braber, T. Dimitrakos, R. Fredriksen, B. A. Gran, S. H. Houmb, Y. C. Stamatiou, and J. Ø. Aagedal. Model-based risk assessment in a component-based software engineering process: The CORAS approach to identify security risks. In F. Barbier, editor, Business ComponentBased Software Engineering, pages 189–207. Kluwer, 2002. ISBN: 1-4020-7207-4. The Honeynet Project. The web page for the honeynet project. http://www.honeynet.org/. 15 December 2004. D. Vose. Risk Analysis: A Quantitative Guide. John Wiley & Sons Ltd., 2000.