Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
Detecting Anomalies in Constructed Complex Systems Christopher Landauer Aerospace Integration Science Center The Aerospace Corporation, Mail Stop M6/214 P. O. Box 92957, Los Angeles, California 90009-2957, USA E-mail:
[email protected], Phone: +1 (310) 336-1361 Kirstie L. Bellman Principal Director, Aerospace Integration Science Center The Aerospace Corporation, Mail Stop M6/214 P. O. Box 92957, Los Angeles, California 90009-2957, USA E-mail:
[email protected], Phone: +1 (310) 336-2191
Abstract It is well-known that complex systems are dicult to design, implement, and analyze. Component-level veri cation has improved to the point that we can expect to produce formal or nearly formal veri cation analyses of all components of a complex system. What remains are the system-level veri cations, which we believe can be improved by our approach to system development. In earlier papers, we de ned the wrapping integration infrastructure, which shows how a little bit of knowledge about the uses of the di erent resources goes a very long way towards using them properly, identifying anomalies and even monitoring their behavior. In this paper, we rst describe our Knowledge-Based integration infrastructure, called \wrappings", then we describe some anomaly detection algorithms originally developed for veri cation and validation of KnowledgeBased systems, and nally, we show how systems organized using wrappings lend themselves to evaluation studies, both oine and online.
1. Introduction Our main interest is the study of coordination among components, agents, and even humans within very large computing systems, without imposing too many design decisions in advance [20], and especially, without precluding any particular architectural style [19]. We have (1) emphasized the system's infrastructure as a way to keep large numbers of disparate com-
ponents coordinated, (2) shown how the infrastructure can be based on explicit meta-knowledge and active integration processes that use the meta-knowledge, (3) introduced a di erent interpretation of interaction with computers, based on a comparison of what we view as the important aspects of many popular programming paradigms, and (4) shown how these notions can be combined to build a framework for heterogeneous system integration. All of these notions derive from our dynamic infrastructure for what we have called Constructed Complex Systems, which are complex heterogeneous systems that are mediated or integrated by computer programs, and in particular, on our \Wrapping" technique, which is based on two notions: (1) explicit, machineinterpretable descriptions of all software, hardware, and other computational resources in a system, and (2) active integration processes that select, adapt, and combine these resources to apply to particular problems. Wrappings are a Knowledge-Based dynamic integration infrastructure for constructing complex heterogeneous software systems. Because the wrappings are intended to de ne everything that the system can do, they can be used to guide system speci cation, and because the wrappings de ne everything that the system can do, they can be used to guide testing. It is well-known that these systems are hard to design, implement, and analyze, because there is no one model, modeling notation, or even modeling style that can be adequate to de ne all aspects of the system [6] [22]. Formal methods have been developed to make the speci cation and analysis of systems more reliable, but
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
1
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
they have not yet reached their expected utility, in part because the intellectual overhead for using them is high, and in part because the mathematical methods available are not powerful enough for complex system models [13]. Component-level veri cation has improved to the point that we can expect to produce formal or nearly formal veri cation analyses of all components of a complex system, but it is still dicult to produce system-level veri cations (For us, veri cation is comparison of a system de nition to a system speci cation, and validation is comparison of the speci cation to the intended behavior). In this paper, we describe the Wrapping approach very brie y, and discuss how it supports detailed analyses of system behavior, using empirical methods.
2. Wrapping The wrapping approach is a computationally re ective, Knowledge-Based integration infrastructure that uses explicit, machine-processable information about resource uses, AND Active processes that use the information to organize resources to apply to posed problems. In this section, we give an overview of the wrapping approach; many more details are elsewhere [9] [10] [12] [14] [15] (and references therein). The wrapping theory has four essential features: 1. ALL parts of a system architecture are resources, including programs, data, user interfaces, architecture and interconnection models, and everything else. 2. ALL activities in the system are problem study, (i.e., all activities apply a resource to a posed problem), including user interactions, information requests and announcements within the system, service or processing requests, and all other processing behavior. We therefore speci cally separate the problem to be studied from the resources that might study it. 3. Wrapping Knowledge Bases contain wrappings, which are explicit machine-processable descriptions of all of the resources and how they can be applied to problems to support what we have called the Intelligent User Support (IUS) functions [2]: Selection (which resources to apply to a problem), Assembly (how to let them work together), Integration (when and why they should work together),
Adaptation (how to adjust them to work on
the problem), and Explanation (why certain resources were or will be used).
Wrappings contain much more than \how" to use a resource. They also help decide \when" it is appropriate, \why" you might want to use it, and \whether" it can be used in this current problem and context. 4. Problem Managers (PMs), including the Study Managers(SMs) and the Coordination Manager (CM), are algorithms that use the wrapping descriptions to collect and select resources to apply to problems. They use implicit invocation, both context and problem dependent, to choose and organize resources. The PMs are also resources, and they are also wrapped. The wrapping information and processes form expert interfaces to all of the di erent ways to use resources in a heterogeneous system that are known to the system [3] [16]. The most important conceptual simpli cations that the wrapping approach brings to integration are the uniformities of the rst two features: the uniformity of treating everything in the system as resources, and the uniformity of treating everything that happens in the system as problem study. The most important algorithmic simpli cation is the re ection provided by treating the PMs as resources themselves: we explicitly make the entire system re ective by considering these programs that process the wrappings to be resources also, and wrapping them, so that all of our integration support processes apply to themselves, too. The entire system is therefore Computationally Re ective [17] [7] [9]. It is this ability of the system to analyze its own behavior that provides some of the power and exibility of resource use. These ideas have proven to be useful, even when implemented and applied in informal and ad hoc ways [18] [5]. Since the process steps that interact with the Wrapping Knowledge Base are themselves posed problems, we can use completely di erent syntax and semantics for di erent parts of the WKB in di erent contexts, and select the appropriate processing algorithms according to context. In particular, whether one writes about one WKB or several is a matter of taste and viewpoint. Even though the WKBs have a variable syntax, there are some common features in the semantics, which we describe next. Each wrapping is a list of \problem interpretation" entries, each of which describes one way in which this
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
2
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
resource can be used to deal with a problem. There may be several problem interpretation entries for the same problem if the resource has many di erent ways of dealing with it. Each of these problem interpretation entries has lists of context conditions that must hold for the resource to be considered or applied. All of these conditions are checked using the current local context. These sets of conditions are important at di erent times; one at resource consideration or planning time, and the other at resource application time. These act as pre-conditions for the application of the resource. The corresponding post-condition is the \product" list of context component assignments, which describes what information or services this resource makes available when it is applied. This means that ANY Knowledge Representation mechanism that: (1) can express the above semantic features, (2) can be queried by problem (in order to match resources to problems), and (3) can be ltered by problem and context information (in order to resolve resource selections), can be used for Wrappings. Many more details can be found in [14] [15], including thorough descriptions of the Wrapping Knowledge Bases and the Problem Managers that are usd in any wrapping-based system.
3. Veri cation and Validation Once we have changed our approach to system development by using the models and the meta-knowledge of wrappings [11], we can apply the usual mathematically formal methods to their analysis. This part of the process is well-known; using wrappings gives us a complementary set of descriptions that we can also analyze. Regardless of how the wrapping knowledge is obtained, it consists of a number of rules for the use and adaptation of resources. The question then arises, \How do we check the correctness of this metaknowledge?", and even more so, \How do we check the consistency of this knowledge with the knowledge in the other wrappings being processed in this system?". We have developed mathematical methods for veri cation and validation (V&V) of KBSs that apply to the wrappings [4]. The theoretical foundation of this work is the very important tenet that rulebases are models, and development and analysis of them should follow sound modeling principles, since building a rulebase is building a model. Some of the genesis and history of our approach, and a placement of it in the context of other V&V research, can be found in earlier publications [1] [8]. In this context, validation refers to the evaluation of system speci cations or behavior according to external desiderata of performance (\is it
the system we intended?"), and veri cation refers to the more mathematically formal process of evaluating a system according to explicit speci cations of behavior (\is the system correct?"). Rulebases whose problems have suciently rich descriptions carry internal redundancies and other relationships that allow some kinds of errors to be caught automatically, simply by a careful examination of the rulebase itself, even in the absence of a complete system speci cation. Our approach is to use empirical analyses, collecting frequency and distribution information about the terms that appear in the expert system, and attempting to identify the strange behavior by looking for anomalies in the distributions. Using this approach, combining this notion with those about global errors and principled methodology [1], we de ned a new set of acceptability principles for rulebases [8]. These principles go beyond concerns over mathematical correctness to considerations of the distribution and simplicity conditions that can signal errors or ineciencies in the rules. The principles we have introduced are called Consistency, Completeness, Irredundancy, Connectivity, and Distribution. This is part of our overall program to develop new principles of correctness, develop criteria illustrating the principles, develop algorithms implementing the criteria, and nally, to develop methods that help correct the errors. The principles are associated with criteria for acceptability in rulebases, and analysis algorithms that can check them. They treat a rulebase as a formal mathematical structure (usually a graph or an incidence matrix), and the analyses show that the mathematical conditions can be checked e ectively, if not necessarily quickly [8]. We have shown these analyses to be very helpful in the early design stages of a system development [4]. This section describes some criteria for acceptability in rulebases, and analysis algorithms that can check them. The criteria are organized according to the principles listed above. They treat a rulebase as a formal mathematical structure (to be de ned in the following sections), and the analyses show that the mathematical conditions can be checked e ectively, if not necessarily quickly.
3.1. Knowledge-based Analysis Tools In order to describe our approach to analyzing rulebases, we start with a careful description of the kind of rulebases we consider. It is a restriction made for the purposes of this discussion, and not a necessary limitation on the techniques. We have applied these techniques to other knowledge-based systems, including sets of equations.
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
3
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
A rulebase is a nite set R of \if-then" rules of the form IF hypothesis, THEN conclusion For example, a rulebase for anomaly detection in spacecraft attitude control might contain a rule such as rule10: IF thruster = on AND thruster-command = o , THEN signal anomaly A clause is either an atomic formula or its negation (for example, a predicate expression such as \thruster=on" above). The set V of variables in a rulebase R is nite. Variables are a little hard to describe, so for now we just note that \thruster" and \thruster-command" above are variables. We will de ne them more precisely shortly. We also make a distinction between the syntactic restriction of having each variable value in the appropriate domain (without regard to the rules) and the semantic restriction that all rules are to be satis ed. A prospective situation is an instantiation of (i.e., an assignment of valid values to) all of the variables, so that each value is in the appropriate domain. This represents a kind of separate syntactic condition for the variable values. A situation is a prospective situation with the further restriction that all the rules are true for all of the situations. This represents is a kind of combined semantic condition for the variable values, if we consider the rules to determine the semantics of the variables. Every variable is considered to be a feature of the situation, with a possibly unknown value in the appropriate domain. The rest of this section explains what this restriction means. It is assumed that there are no unde ned values. Each variable v is considered to be a function applied to situations, so for a situation s, the expression v(s) denotes the value of the variable v in situation s. More generally, for any expression e over a set W of variables contained in V , the expression e(s) denotes the value of the expression e in situation s. Rules are implicitly universally quanti ed over situations. A variable v in the rulebase is a xed component selection function v applied to a logical variable representing the situation s. There are no explicit quanti ers, so all situation variables are free in the expressions. The set of situations is therefore a subset of the prospective situations, which is the Cartesian product of all of the variable domains; however, the particular subset is not precisely known, since it is limited by the rules in the rulebase to only those elements of the Cartesian product that satisfy the rules (because the rules de ne the situations). The rules may, for
example, de ne connections between variables that allow some of the variables to be computed from others. The Cartesian product will occasionally be called the Problem Space, to distinguish it from the set of situations, which is called the Solution Space or Situation Space. Therefore, the syntactic restriction of having each variable value in the appropriate domain de nes the prospective situations, and the semantic restriction that all rules are to be satis ed de nes those prospective situations that are situations. A rulebase is applied to a situation to compute some variable values (not to set the values, but to nd out what the values are), so that a situation has both provided variable values (\input" variables) and derived variable values, some of which are internal (\intermediate" or \hidden" variables) and some of which are displayed (\output" variables). It is further assumed that the variable values not speci ed by the input are de ned but unknown, and that the rulebase is expected to compute them (or at least to compute the output variable values). We can model other styles of expert system this way also, in terms of access to parameters and processing steps. We have concentrated on rulebases of this form for convenience in description.
3.2. Incidence Matrices and Graphs We are most interested here in the case in which we have no models or other external criteria for correctness in the rulebase, since this is the most common case, and since it is the hardest case. We cannot in that case hope to prove very much, since we quickly run into undecidability problems, but we can identify places that are \strange" (for various de nitions of \strage"). Then the rulebase designer can decide whether those anomalous conditions belong in the rulebase or not. We are interested therefore in examining the occurrence patterns of various elements in a rulebase, because they provide a very gross model of the rules and rule interactions, and because they lead the use of some wellknown mathematical techniques. We start with a tool that allows us to record precisely what we mean by an occurrence and an occurrence pattern. It is a common tool in combinatorial analyses called an incidence matrix [23]. It also lets us de ne a graph that allows a di erent formal analysis of the occurrence patterns. There is a standard equivalence between square incidence matrices and graphs, given by making a vertex for each row or column, and an edge for each nonzero entry in the matrix (sometimes the edge is labeled with the nonzero entry). Conversely, the matrix has a row and column for each vertex, a zero entry for each non-edge, and a nonzero en-
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
4
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
try for each edge. The nonzero entry is usually 1 for unlabelled graphs, and it is usually the label for edgelabelled graphs. The incidence matrix is a good numerical representation of the occurrences of elements in the rulebase (we still haven't said what those elements are), and allows natural numerical summaries and analyses (correlations, etc.). The graph is a good topological representation of the occurrences of elements in the rulebase, and allows natural combinatorial and topological analyses (connectivity, etc.). This paper will not discuss in detail the di erent kinds of graphs that can be formed from a rulebase. More details can be found in [8]. The simplest incidence matrix of a rulebase counts the number of occurrences of variables in rules. The counting incidence matrix RV is a matrix indexed by R V , with RV (r; v) = no. instances of variable v in rule r; For example, the rule listed above would have a row for \rule10", and columns for \thruster" and \thrustercommand", with both entries 1. There are also incidence matrices RC for clause occurrences in rules and CV for variable occurrences in clauses. Many combinations of incidence matrices have signi cance for analyzing a rulebase. From this matrix RV , higher products can be computed, for example, giving the number of pairs of variables that appear in the same rule, or the number of pairs of rules that contain the same variable. For example, (RV RV tr )(q; r) = number of variables in common between rule q and rule r, (RV tr RV )(v; w) = number of rules containing both variable v and w. Two simple extensions of RV that are used in the example analysis separate the rules into hypotheses and conclusions. The matrix V H has a row for each variable and column for each rule, with entries that count the number of occurrences of the variable in the rule hypotheses, and the matrix V C has a row for each variable and column for each rule, with entries that count the number of occurrences of the variable in the rule conclusions. In particular, we have RV tr = V C + V H; where we write M tr for the transpose of a matrix M (interchange rows and columns). Many other combinations of incidence matrices have signi cance for analyzing a rulebase. Furthermore, square matrices are easier to consider as graphs, since we can identify rows and columns. For example,
(V H tr V H )(q; r) = number of variables in common between rule q and rule r hypotheses, (V C V C tr )(v; w) = number of rules containing both variable v and w in their conclusions, (V C tr V H )(q; r) = number of variables in common between rule q conclusion and rule r hypoth-
esis, Many other variations exist also. Among these products and others, the matrix (V C tr V H ) is particularly relevant for studying the inference system. Several more detailed incidence matrices are also useful for rule analyses [8] [4], such as matrices that separate the roles of variables into reading and writing, and those that consider clauses. There are formal de nitions of reading and writing, but they will not be needed here. The intuitive notions of using a value and setting a variable will suce. A variable v is read by a rule r if v occurs in r, either in hyp(r) or in conc(r), in such a way that its value is required. Not every occurrence in conc(r) is a read access, but every occurrence in hyp(r) is (except when the hypothesis expression can always be evaluated without using the value of v). A variable v is written by a rule r if v occurs in conc(r) in a way that allows the value to change. For example, a variable used only as the source of a value is not written, but a predicate that compares one variable to another, without specifying which one changes or how they change, is assumed to write both. It should be noted here that \write" only means that a new value becomes known to the system. There is no change in the value, only in what the system knows about it. The variable read matrix Rd is indexed by R V , as is the variable write matrix Wr. For a rule r and variable v, the boolean access matrices are de ned by rule r reads variable v; Rd(r; v) = 10 ifotherwise ; and rule r writes variable v; Wr(r; v) = 10 ifotherwise (so Wr need only consider conc(r), but Rd needs both hyp(r) for condition testing and conc(r) for value sources), and the counting access matrices use the number of read and write instances of v in r instead of using only their existence. For two variables v and w, (Rdtr )(Wr)(v; w) is nonzero i there is a rule r which reads v and writes w. For two rules q and r, (Wr)(Rdtr )(q; r) is nonzero i there is a variable v which is written by q and read
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
5
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
by r (if the counting access matrices are used, then the products also count how many such rules or variables there are). These two product matrices are clearly related to a kind of required ordering between variables and between rules. Since we assume that the variable values do not change, they must be written before they are read, unless they are provided as input. We can certainly prove some things about the original rulebase using the matrices, such as the nondeterminability of a variable value (since no rule sets it), or some kinds of deadlock in the rules (since some rules that set a value can only occur after rules that use it), but since in general the matrix is a very loose model, the results are fairly sparse and weak. Instead, we usually have to fall back on empirical methods, with a corresponding retreat from proofs.
3.3. Association Matrices An association matrix is a covariance matrix computed from occurrence patterns across a set of possible locations. The counting incidence matrix product, de ned by (RV )(RV tr )(q; r) = commonality of q and r; which is the number of variables v common to rules q and r, counts variables in common to rules, measuring the occurrence pattern of a rule according to the variables it contains. Then the correlations can be computed from the covariances, in the usual way:
Cor(q; r) = p Cv(q; r)=(Sd(q) Sd(r)); Sd(q) = Cv(q; q); Cv(q; r) = (RV RV tr )(q; r)=jV j , Av(q) Av(r); X Av(q) = (vars v 2 V ) RV (q; v)=jV j: Here, the q row of the counting incidence matrix RV is the occurrence pattern for rule q, so Av(q) is the average number of occurrences of each variable in rule q, and Sd(q) is the standard deviation of the numbers. There is no random variable here, so there is no point in using the \sample standard deviation". The correlation is a measure of similarity between rules, as measured by the variables in them. The correlation value is 1 if and only if the two rules use exactly the same variables with the same frequency of occurrence of each variable. It will be negative, for example, when the two rules use disjoint sets of variables, and -1 in rare cases only (not likely in a rulebase). Similarly, the matrix product (RV tr )(RV ) counts rules that occur in common to two variables for each
pair of variables, measuring the occurrence pattern of a variable by the set of rules containing it. Correlations are computed as before. Other incidence matrices for variables in clauses and clauses in rules can also be used in this way. The use of correlations is in detecting unusual ones. If clause b almost always occurs with c, then something should be noted when they do not occur together. If variable v always occurs with w, then there may be a good reason for combining the variables. There should also be sucient justi cation for unusual correlations or distinctions. These uses of association and correlation matrices come down to one question (expressed here only for variables, but equally applicable to clauses or rules or other constructions): If v and w are highly correlated, then why are they di erent? Since each covariance matrix above is symmetric and positive semi-de nite (as are the corresponding correlation matrices), one can consider the matrix to be a \similarity matrix" that indicates which elements are similarly distributed. Then the similarity measurements contained in the correlation matrix can be used in a cluster analysis [21]. Clustering methods are simple, and can give useful information, since the clusters of rows in RV (for example) are sets of rules that use (roughly) the same variables. More information on this approach can be found in [8].
3.4. Criteria for Rulebase Correctness This subsection describes some principles of rulebase correctness, and ways to test them for a particular rulebase. There is no description of how to determine whether or not to test the principles, since that decision is rulebase dependent. A principle of rulebase correctness is a condition on a set R of rules that is required for the rulebase to be reasonable in some incompletely de ned sense. This notion is not the same as a principle of modeling a process or a system by rules (that step is hard [1]). Rather, it is a notion of how rules t together into a rulebase. In order to understand the range of possible criteria for rulebases, di erent uses of the rulebase must be considered. The major uses considered thus far are as a speci cation, i.e., a de nition of classes of situations, or as an evaluation, i.e., a determination of the proper class for a given situation. These styles of application require two kinds of consistency. The speci cation aspect of a rulebase requires static (or non-procedural) consistency of the rules as
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
6
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
logical de nitions. The evaluation aspect of a rulebase requires dynamic (or procedural) consistency of the rules as algorithm de nitions (remember, the inference engine is considered to be part of the rulebase). Static analysis refers to the examination of the rules as separate symbolic expressions, without stringing them together, dynamic analysis refers to the interactions of the rules during inference, logical analysis refers to the form of the rules as mostly uninterpreted expressions, and statistical analysis refers to the values of the variables. The ve principles are as follows: Consistency (no con ict), Completeness (no oversight), Irredundancy (no super uity), Connectivity (no isolation), and Distribution (no unevenness). These principles are implemented by many criteria for rulebase correctness. The criteria are separated into classes, according to the principles they implement. Analyzing a rulebase for these principles involves implementing mathematical and computational criteria that check for the application of each principle. The Consistency criteria, which address the logical consistency of the rules, can rightly be considered as \correctness" criteria. The Completeness and Irredundancy criteria, which preclude oversights in speci cations and redundancy in the rules, are more like \reasonableness" criteria for the terms in the rules. The Connectivity criteria, which concern the inference system de ned by the rules, are like completeness and irredundancy criteria for the inference system. Finally, the Distribution criteria are \esthetic" criteria for simplicity of the rules and the distinctions they cause, as well as the distribution of the rules. The rst three principles, Consistency, Completeness, and Irredundancy, are not discussed in detail in this paper, since they are relatively easy to explain and have appeared in other rulebase V&V research. The Connectivity criteria were discussed earlier [8], so they too will be described only brie y here. The rest of this section discusses the simplest of the corresponding criteria, without much detail. The Distribution and Simplicity principles are discussed in detail in [8]. The Consistency principle leads to criteria that involve some kind of lack of con ict among rules. The situations should be well de ned, as should all the interesting variable values. The criteria will not be listed here, as they correspond to easy syntactic checks. For example, if a rule also checked \thruster = on" and
\thruster-command = o ", but set \signal ok" instead of \signal anomaly", then that rule is inconsistent with \rule10". The Completeness principle leads to criteria that involve some kind of universal applicability of the rulebase. Defaults are usually used to guarantee certain kinds of completeness. All detectable places where defaults will be used should be signaled, since some of these places may only indicate undesired incompleteness in a rulebase, instead of places that are expected to be xed by the use of defaults. These criteria will also not be listed here. The Irredundancy principle leads to criteria that insist that everything in the rulebase is there for some good reason. The variables make a di erence, the rules make a di erence, and nothing is extraneous. There are good reasons to note redundancies as more dangerous than mere ineciency; they are often the result of incomplete models or modeling criteria [1].
4. Discussion The analyses we describe above can clearly be applied to Wrapping Knowledge Bases [4], by translating the wrappings into \if-then" type rules. In that way, we can learn many things about the overall shape of the system: whether certain resources can ever be used, what classes of resources use what other classes, etc.. Remember, we are most interested in systems that are so large, heterogeneous, and dynamic, that we do not necessarily know all of the components in advance, so these analyses cannot be complete decision procedures for the corresponding questions. Of course, it is hard to analyze a system in which we do not know all of the components, but we can imagine that there might be a planner whose problem decompositions depend on being able to nd certain computational services, or a component whose task it is to nd certain services. Then, since we have criteria for what these components are looking for, we can at least perform some rudimentary analyses. There are a few kinds of problems with constructed complex systems that we have not discussed: mainly communication issues (are the right bytes getting to the right places?) and timing issues (are there any important timing anomalies?). We can study both of these using a wrapping infrastructure also, but it requires some more careful modeling and meta-modeling. We can imagine \shadow" resources that are applied only in a \simulation" mode, but with exactly the same selection and application criteria. Their purpose is to simulate the components they shadow, performing only some of the calculations, but taking, as accurately as
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
7
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
possible, the same amount of time. Then simulation mode can be used to try to nd timing anomalies and danger spots. The system can also be driven using a simulation clock in the usual discrete event simulation manner. Similarly, at run time, the complete mediation of the wrapping infrastructure means that various kinds of correctness or partial correctness criteria can be applied as part of the resource selection process. This means that we can insert instrumentation, data diversion, security policy validation, or any other kind of inter-component marking procedure between any component invocations. This ability gives us a great exibility in studying the behavior of complex distributed systems. We believe that using a wrapping-based architecture will help to make system-level speci cation more transparent and analyzable, and system-level veri cation and testing more reliable.
References
[6] [7] [8]
[9]
[10]
[1] Kirstie L. Bellman, \The Modelling Issues Inherent in Testing and Evaluating Knowledgebased Systems", pp. 199-215 in Chris Culbert (ed.), Special Issue: Veri cation and Validation of Knowledge Based Systems, Expert Systems With Applications Journal, Volume 1, No. 3 (1990) [2] Kirstie L. Bellman, \An Approach to Integrating and Creating Flexible Software Environments Supporting the Design of Complex Systems", pp. 1101-1105 in Proceedings of WSC '91: The 1991 Winter Simulation Conference, 8-11 December 1991, Phoenix, Arizona (1991); revised version in Kirstie L. Bellman, Christopher Landauer, \Flexible Software Environments Supporting the Design of Complex Systems", Proceedings of the Arti cial Intelligence in Logistics Meeting, 8-10 March 1993, Williamsburg, Va., American Defense Preparedness Association (1993) [3] Kirstie L. Bellman, April Gillam, Christopher Landauer, \Challenges for Conceptual Design Environments: The VEHICLES Experience", Revue Internationale de CFAO et d'Infographie, Hermes, Paris (September 1993) [4] Kirstie L. Bellman, Christopher Landauer, \Designing Testable, Heterogeneous Software Environments", pp. 199-217 in Robert Plant (ed.), Special Issue: Software Quality in KnowledgeBased Systems, Journal of Systems and Software, Volume 29, No. 3 (June 1995) [5] Dr. Kirstie L. Bellman, Captain Al Reinhardt, USAF, \Debris Analysis Workstation: A Modelling Environment for Studies on Space Debris",
[11]
[12]
[13]
[14]
Proceedings of the First European Conference on Space Debris, 5-7 April 1993, Darmstadt, Germany (1993) Richard Bellman, P. Brock, \On the concepts of a problem and problem-solving", American Mathematical Monthly, Volume 67, pp. 119-134 (1960) Gregor Kiczales, Jim des Rivieres, Daniel G. Bobrow, The Art of the Meta-Object Protocol, MIT Press (1991) Christopher Landauer, \Correctness Principles for Rule-Based Expert Systems", pp. 291-316 in Chris Culbert (ed.), Special Issue: Veri cation and Validation of Knowledge Based Systems, Expert Systems With Applications Journal, Volume 1, No. 3 (1990) Christopher Landauer, Kirstie L. Bellman, \Knowledge-Based Integration Infrastructure for Complex Systems", International Journal of Intelligent Control and Systems, Volume 1, No. 1, pp. 133-153 (1996) Christopher Landauer, Kirstie L. Bellman, \Integration Systems and Interaction Spaces", pp. 161-178 in Proceedings of FroCoS'96: The First International Workshop on Frontiers of Combining Systems, 26-29 March 1995, Munich, Germany (March 1996) Christopher Landauer, Kirstie L. Bellman, \Model-Based Simulation Design with Wrappings", pp. 169-174 in Proceedings of OOS'97: Object Oriented Simulation Conference, WMC'97: 1997 SCS Western Multi-Conference, 12-15 January, Phoenix, SCS International (1997) Christopher Landauer, Kirstie L. Bellman, \Wrappings for Software Development", pp. 420429 in 31st Hawaii Conference on System Sciences, Volume III: Emerging Technologies, 6-9 January 1998, Kona, Hawaii (1998) Christopher Landauer, Kirstie L. Bellman, \Where's the Math? The Need for New Mathematical Foundations for Constructed Complex Systems", to appear in Proceedings ICC'98: the 15th International Congress on Cybernetics, 2327 August 1998, Namur, Belgium (1998) Christopher Landauer, Kirstie L. Bellman, \Generic Programming, Partial Evaluation, and a New Programming Paradigm", Paper etspi02 in 32nd Hawaii Conference on System Sciences, Track III: Emerging Technologies, Software Process Improvement Mini-Track, 5-8 January 1999, Maui, Hawaii (1999); revised and extended version in Christopher Landauer, Kirstie L. Bellman, \Generic Programming, Partial Evaluation, and a New Programming Paradigm", Chapter 8, pp. 108-154 in Gene McGuire (ed.), Soft-
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
8
Proceedings of the 33rd Hawaii International Conference on System Sciences - 2000
[15]
[16]
[17]
[18]
[19] [20]
[21] [22]
[23]
ware Process Improvement, Idea Group Publishing (1999) Christopher Landauer, Kirstie L. Bellman, \Problem Posing Interpretation of Programming Languages", Paper etecc07 in 32nd Hawaii Conference on System Sciences, Track III: Emerging Technologies, Engineering Complex Computing Systems Mini-Track, 5-8 January 1999, Maui, Hawaii (1999) Christopher Landauer, Kirstie L. Bellman, April Gillam, \Software Infrastructure for System Engineering Support", Proceedings AAAI'93 Workshop on Arti cial Intelligence for Software Engineering, 12 July 1993, Washington, D.C. (1993) Pattie Maes, D. Nardi (eds.), Meta-Level Architectures and Re ection, Proceedings of the Workshop on Meta-Level Architectures and Re ection, 27-30 October 1986, Alghero, Italy, NorthHolland (1988) Lawrence H. Miller, and Alex Quilici, \A Knowledge-Based Approach to Encouraging Reuse of Simulation and Modeling Programs", in Proceedings of SEKE'92: The Fourth International Conference on Software Engineering and Knowledge Engineering, IEEE Press (June 1992) Mary Shaw, David Garlan, Software Architecture: Perspectives on an Emerging Discipline, Prentice-Hall (1996) Mary Shaw, William A. Wulf, \Tyrannical Languages still Preempt System Design", pp. 200211 in Proceedings of ICCL'92: The 1992 International Conference on Computer Languages, 20-23 April 1992, Oakland, California (1992); includes and comments on Mary Shaw, William A. Wulf, \Toward Relaxing Assumptions in Languages and their Implementations", ACM SIGPLAN Notices, Volume 15, No. 3, pp. 45-51 (March 1980) R. Sibson, \SLINK: An Optimally Ecient Algorithm for the Single-Link Cluster Method", Computer J., Vol. 16, pp. 30-34 (1973) Donald O. Walter, Kirstie L. Bellman, \Some Issues in Model Integration", pp. 249-254 in Proceedings of the SCS Eastern MultiConference, 2326 April 1990, Nashville, Tennessee, Simulation Series, Volume 22, No. 3, SCS (1990) Wilson, Robin J., Introduction to Graph Theory, Longman, London (1972)
0-7695-0493-0/00 $10.00 (c) 2000 IEEE
9
