Formal Methods for Modeling Socio-technical Innovation Between Adversaries Russell Cameron Thomas Department of Computational Social Science Krasnow Institute of Advanced Study, George Mason University
[email protected]
Abstract—Information security is often called an ‘arms race’, but little is known about the co-evolutionary dynamics of innovation. To facilitate such research, we define two formal methods that can be executed by computational agents in a multi-agent system. First, we formalize the definition of capabilities and business models as a ‘viable system’. We generalize it to encompass technological, sociological, and economic aspects of a system, which together comprise its capabilities. Second, we formalize the method of pattern language. This allows agents to generate and reason about alternative solutions. This expands the space of possibilities, and thus simulates a form of inventiveness. We conclude with an example of the methods applied to email spam and antispam defense. Keywords: information security; evolutionary arms race; coevolution; pattern language; capability models; business models; email spam.
I.
INTRODUCTION
Information security1 has been commonly viewed as a rivalry between attackers and defenders, and as being characterized by an “evolutionary arms race” where each side has incentives to continually create in new innovations to overcome the opponent’s capabilities. This, in turn, requires continual replacement of old solutions with new. In doing so, neither side gains a lasting advantage (i.e. the Red Queen effect) [1]. If this is true, then it will be important at a policy level to understand the dynamics of the evolutionary arms race in order to determine the intermediate and long-term trajectory of the game, and which policy decisions will swing the trajectory more in favor of defenders and for the greatest social welfare. This is especially true for government agencies that sponsor research and development (R&D), critical infrastructure companies, and regulatory bodies. However, little research has been done regarding the evolutionary dynamics of information security. Coevolutionary innovation can occur over several time scales – from the scale of major technology and business innovations (months to years), to the time scale of an attack 1
We focus on information security but the same methods could apply to any other adversarial setting where the ‘business model’ of the adversaries is a significant arena of innovation. Examples include cyber-physical and physical security, organized crime, tribal feuds, and perhaps some aspects of national security.
campaign (hours to months) or a single attack (milliseconds to days). Thus, it would be very useful to have a unifying framework for co-evolutionary innovation across many time scales. One major obstacle to research is the lack of formal methods for modeling innovation in general and coevolutionary, adversarial innovation in particular. We believe that formal modeling of innovation processes will make a significant contribution by helping defenders increase their agility in defending against adversaries that innovate rapidly. Formal modeling of the innovation facilitates simulation, including multi-agent systems (MAS). (The agents in MAS represent intelligent actors or groups of actors, or formal organizations. They can also represent autonomous technical systems or sociotechnical systems.) Simulations can be used to explore scenarios and to challenge conventional wisdom. They can help analysts explore the possible paths of innovation and to devise advantageous strategies that would ‘change the game’ in their favor. Operationally, simulation systems can integrate intelligence regarding attacker and defender capabilities. This would reveal gaps and possible consequences. This would then provide valuable guidance in system engineering, risk analysis, resource deployment, metrics and leading indicators, incentive systems, institution design, governance, regulatory mandates, etc. A. Formalizing methods of innovation A useful starting place to model innovation is to study existing informal methods used by business and engineering practitioners. In this paper we consider three methods in particular: • Capabilities [2] [3] [4] • Business Models [5] • Pattern Language [6] These informal methods have the advantage that they have been tested, at least conceptually, by a community of practitioners who are innovating in real-world settings. Informal methods help the practitioners share their experience and what they have learned, and they also reinforce certain viewpoints and ways of acting. We can’t be sure if they are the best methods or even that they are widely used. The best we can say is that they are each sufficiently plausible for their given application and purposes. This is good enough for our purposes. But lack of formalization makes it hard or impossible to implement them in a computational model. Another
obstacle is that they are not easy to use together because they operate on different levels of abstraction and define the system of interest in ways that are incommensurate. This presents an opportunity to formalize them so they can be used together in computational models aimed at simulating realistic scenarios. This paper presents two formalizations for modeling socio-technical innovation in adversary multi-agent systems. First, the informal concepts of capabilities, dynamic capabilities, and business models are combined into a formal model of ‘viable systems’. This simulates a type of holistic thinking where the various components and aspects are brought together into a whole system and considered according to its potential for performance in various environments. Second, the Pattern Language is generalized and formalized to provide an agent with the capability to generate and reason about alternative architectural solutions. This simulates a type of creative inventiveness. In future work, we will generalize and formalize a method of inventive problem solving to allow simulation of heuristic search in the design space. II. A FORMAL MODEL OF VIABLE SYSTEMS A system is said to be viable if the people in the system have good reason to believe that the system will perform well in the future, in the face of a challenging and selective environment. This concept of ‘viable system’ is close to the evolutionary biology notion of ‘fit’ and ‘fitness’. In contrast, ‘viability’ doesn’t imply that the system is optimal in the sense used by the engineers or economists. Viability is achieved and maintained through constant adjustment, anticipation, and a bit of imagination, not by any process of optimization or optimal selection. Agents in the system judge it to be ‘good enough’ and perhaps ‘better than most’. Thus, a ‘viable system’ is always a tentative judgment by the people in the system and can never be fully falsified because it’s not possible to rerun history to explore all the counterfactual paths. Because the future environment is hard or impossible to predict in this setting, people adopt heuristics that allow them to judge viability of alternatives ex ante and also to apply past experiences to new opportunities and environments. B. Business models In this paper we adopt the concept of ‘business model’ as a viable system heuristic for commercial enterprises, and adapt it to adversarial settings. In the following, we draw primarily on Osterwalder [5]. In a commercial marketplace, a business model can be viewed ontologically as the interaction between three semantic entities – a target customer, a value proposition tuned to that target customer, and a set of capabilities that enable fulfillment of that value proposition (Figure 1).
Figure 1. A simple business model ontology for commercial systems
Figure 2. The simple ontology extended to one-way adversarial systems
However, in the adversarial setting, we need to modify this ontology to reflect the coupling between attackers and defenders, as shown in Figure 2. What was previously called ‘value proposition’ becomes, for attackers, their exploitation scheme to extract value from their target. While a commercial value proposition is attractive to the target customer, exploitation is repugnant to the target entity. Conversely, the ‘value proposition’ of the defender is their protection scheme. Instead of providing ‘value’ to the target entity as in the commercial setting, protection provides risk reduction – roughly, it reduces the possible future downside costs of successful attacks. (The term “Asset Owner” is chosen for the domain of information security. Also, we assume the attacker is motivated by profit. Other motives could be incorporated without significant changes.) This business model ontology could be further extended to include a ‘Counter-attacker’, which could be a law enforcement agent or similar. But the extension in Figure 2 is sufficient to set the stage for co-evolutionary innovation. Osterwalder [5] expanded this into a complete and formal ontology for commercial business models. We adapt his business model ontology to one-way adversarial systems in the Fig. 3 below.
ones. That is why we need to add formal descriptions of capabilities and, later, the pattern language.
Figure 3. Attacker business model ontology
For attackers, major ontology categories are: • Asset Interface: What assets does the attacker go after? How does it get access to those assets, and how does the attacker stay shielded (i.e. hide their actions and intent, cloak their identity, deflect counter-attacks, etc.) • Attack: What are the primary modes of attack and modes of exploiting target assets? This roughly defines the ‘business segment’ of the attacker. • Infrastructure Management: How the attacker efficiently performs operations, processes, logistics, management, learning, and innovation, and in what organizational form? • Financial Aspects: How does the attacker monetize the results of their attacks? What is their cost structure? How is the business model sustainable? The defender’s ontology would have the same categories, but would be defined slightly differently. For example, in Financial Aspects, the defender’s equivalent to “revenue” is “loss”, and their equivalent to “profit” is “risk”. (Detailed description of the ontology of both business models is beyond the scope of this paper.) Osterwalder [5] created a markup language for his ontology called BM2XL. It can easily be adapted to the one-way adversarial system following the approach used above to adapt the top-level ontology. This formal ontology provides modelers with a way to define the computational agents and objects in the adversarial system. But it doesn’t provide any way to for agents to understand how a given business model might perform in a given situation and it doesn’t give the agents any method for modifying business models or creating new
C. Capabilities Broadly speaking, ‘capabilities’ are how anything gets done in the face of some resistance, difficulty, or constraints, with an eye toward efficient and effective performance. Capabilities require resources, which could be in the form of physical assets or inputs, physical energy, activity of people, data or information, and time. Moreover, capabilities require knowledge and skill to put into action, and this is usually developed through experience, also known as learning-by-doing. Wallin [7] describes capabilities as repeatable patterns of action in the use of assets to create, produce, and/or offer products and services to the market. Capabilities can be performed internally or they are outsourced to partners. In information security as elsewhere, information and communication technology (ICT) has made it possible for both attackers and defenders to "unbundle" and outsource capabilities and resources that do not belong to their core competencies [8]. Abstractly, capabilities can be seen as defining the upper limits of feasible performance or results for a given set of resources and skills. But putting capabilities into action may take time and effort to ramp up. Thus, capabilities define the scope of strategic options and the frontier of possibilities, and not the actual performance at any specific time. The concept of capabilities is used widely in management science and related fields. In the Resourcebased Theory of the firm (RBT) [9], capabilities that are unique and hard to replicate are seen as the major source of sustainable competitive advantage and profitability. A firms’ capability to innovate its capabilities has been called ‘dynamic capabilities’ [4], which is a central theme in evolutionary economics. The concept of capabilities has also been used extensively in military planning [10]. From a design point of view, capabilities can be seen as a type of external specification. Gero [11] defined the Function-Behavior-Structure (FBS) ontology for designed systems that can be adapted to formalize capabilities. Using FBS we define a capability C as:
C = (F, B, S, K, X, V), where: F = Functions, meaning the purposes, goals, and intentions that can be achieved through the capability. Most often these are expressed as a list of relevant performance dimensions or states and some relational operators on those dimensions/states. B = Behaviors, meaning the modes of action, response, procedure, or performance that support the Functions F. Behaviors are distinguished between actual and expected behaviors (Bs, Be).
S = Structure, meaning the elements and components of the capability and their relations. These could include resources, energy, time, money, people, information, technologies, processes, timeframes, dependencies, etc. Structure S can be represented as UML diagrams or their formal equivalents. K = Knowledge or skills needed to activate the capability. This could be the domain knowledge or procedural knowledge. A score on an ordinal scale of skill levels could represent this, for example. X = Context, meaning the associations between this capability and other capabilities. Context partially determines when a capability should be activated. V = Valuation, meaning the functions that enable the agent to make value judgments regarding the combined effects of Functions F, Behaviors B, and Structures S. FBS allows computational agents to reason about artifacts [12], including capabilities as we have formally defined them. For example, two existing capabilities can be combined to create a third by taking the union of their FBS characteristics and resolving conflicting elements. Of course, the process of artfully resolving conflicting elements is the essence of professional design and engineering practice. FBS does not, by itself, automate this process, but it does give us away to formally reason about the possibilities and outputs of design and engineering because FBS is descriptive. For example, two capabilities can be evaluated to be functionally equivalent if their functions F are the same, even if they exhibit a different structure S or behaviors B. A given capability can be made easier to use by low-skilled agents by reducing the knowledge K required to put it to use. Our formalization of capabilities within a business model can be mapped to the NK model of fitness landscapes used by Kaufman [12] and others to model technology evolution. In an NK Landscape, a technology is described as a bit string of N characteristics. Each characteristic has a fitness value, but each characteristic also depends on K of its neighbors. In our formalization, the total number of functions F in all the capabilities roughly corresponds to the ‘N’ characteristics in the NK model. The structure S and context X of all the capabilities will define the dependencies between functions F and between capabilities C, and thus corresponds to ‘K’ in the NK model. Therefore, our formalism will permit creation and search of fitness landscapes in a similar fashion as NK landscape research. Specifically, functions F define the N characteristics in Kaufman’s genome. The structure S defines (locally) the parameter K, which in Kaufman’s model stands for the number of characteristics that affect the functioning of each characteristic. The main difference is that NK landscapes are generally pure abstractions, whereas our formalism is grounded in a specific domain
and has specific referents. Another major difference is that NK landscapes are static in the sense that the bit strings and length don’t change during model execution. Our formalization allows the space of possible designs to change through the method of pattern language, described below. Thus far, we have defined capabilities defined solely in terms of ontology, which is declarative, static, and descriptive. We also need them to be dynamic and executable. For this we need to map capabilities to their performance characteristics, i.e. the range of feasible performance that each capability provides in a given circumstance. (A performance characteristic or performance dimension is any quantitative or qualitative state change of interest to the agents. In our example in section IV, spammers are interested in performance dimensions of cost, volume of email sent per period, delivery rate, consumer response rate, and monetary value per response.) We accomplish this through contextual affordances. Adapting the definitions from Papasimeon [14], we define contextual affordances as relations between agents and their available capabilities and their environment, given the agent’s context and history. The dependence on context and history makes affordances path-dependent, which in turn points to the essential role of knowledge and learning in the innovation process. A contextual affordance is implemented as an algorithm that accepts signals and context as inputs, along with one or more performance queries. The algorithm then outputs a range of feasible performance along each dimension, and also a monotonic cost function for each performance dimension. Each capability available to an agent is ‘tagged’ with contextual affordance algorithms so they can be queried by agents [14]. This modeling method assumes that the performance functions and learning functions are reasonably well understood, at least in terms of their overall shape and boundary values. Our aim is to explore the range of possible affordances, and therefore we present a method for representing families of performance and learning functions via approximations rather than detailed representation of any single function. This should be sufficient for the goals of the simulation systems, which mostly involve exploring scenarios and the space of possibilities rather than forecasting or making specific predictions.
Formally, a contextual affordance algorithm is defined as follows: A 0 : the set of all contextual affordance algorithms available to an agent at t = t 0
A ( i ) ∈A 0 : a single contextual affordance algorithm L ( i ) : a family of learning-cost functions
l ( t, x ) ∈L : a learning-cost function of time and x t : time dimension in a simulation run V : the set of all variables used in R and L VA ⊆ V : the set of all variables use in affordance algorithm A X : the set of all performance dimensions X A ⊆ X : the set of performance dimensions influenced by A X1 : a query from an agent to A regarding X x ∈ X : a value in performance dimension X α ⊆ V : a vector of agent inputs (e.g. resources, energy, man/hrs of time, signals, etc.) β ⊆ V : a vector defining the agent's context φ : a function shape parameter for R
)
1 : function getAffordance( A, α , β , X1 ) R ← {} L ← {}
4:
for all v ∈VA do v ← assign (α , β )
5:
/ / assigns default value of no inputs or constraints if X1 ∈ X A then
9:
define M : Ri , L j k ; i = [1, n], j = [1, n], k = [1, r]
4: 5:
maxBenefit ← 0 preferredAction ← {}
6: 7:
for all A ∈A 0 do for all X ∈X do
8:
(
)
M ← getAffordances( A, α , β , X )
9:
expectedBenefit ← evaluate(α , β , M , X )
10 : 11 : 12 :
if expectedBenefit > maxBenefit then maxBenefit ← expectedBenefit if maxBenefit > threshold then
preferredAction ← assignAction ( M , X )
13 : 14 : 15 : 16 : 17 : 18 : 19 :
end if end if end for if preferredAction = {} then preferredAction ← exploreAlternativeDesigns end if return ( preferredAction )
The evaluate(•) function is where the fitness landscape is queried. In the case of adversarial systems, the fitness landscapes between attackers and defenders will be tightly coupled. Therefore, the fitness (“expected benefit” in all performance dimensions) of an attack capability will be a function of the fitness of relevant defense capabilities. This is a significant modeling challenge, but discussion is beyond the scope of this paper. In future work we will be developing this formally and exploring alternative implementation methods.
The functions R(•) and L(•) could take many forms: real-valued functions, differential equations, stochastic functions, state machines, or table lookup systems. The choice will depend on the domain, research goals, and trade-offs in model complexity. See section IV for an annotated example. The following pseudo code describes a general contextual affordance algorithm, followed by the pseudo-code for how agents evaluate the affordances:
7:
3:
A (α , β , X1 ) → ( R(φ ), L(λ ))
(
8:
constants threshold, exploreAlternativeDesigns
20 :
R : (α , β , X1 , X A , φ ) → X x ; x ∈[ x, x], X1 ∈ X A x L : α , β , X x , λ → lλ ( t, x ) ; x ∈[ x, x]
6:
2:
λ : a function shape parameter for L and l ( t, x )
x
2: 3:
1 : function evaluateAffordances(α , β , A 0 , R, L )
φ ← setShape(V )
R ← generatePerformanceRange(V, φ )
λ ← setShape(V, R )
L ← generateCostFunction (V, R, λ )
10 : end if 11 : return (R, L) 13 : end function
21 : end function
III. A FORMAL PATTERN LANGUAGE FOR INNOVATION Alexander [6] introduced a design and analysis method called Pattern Language in the field of architecture and design. Patterns represent generalized knowledge regarding the organization and configuration of a system as a whole and its components at several levels of description. Each pattern “describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice” [6]. Because they are generalized, and, to a degree, only partially specified, they must be contextualized in a specific circumstance. A pattern language is a generative grammar of patterns that can be used by designers to explore the space of designs, or to analyze existing designs regardless of their origins. “Each pattern makes reference to other patterns and describes when [they] might be more appropriate; one pattern is often expressed using the language provided by
other patterns.” [15] Every expression in the pattern language is a hierarchical network of patterns. Patterns can be functional – describing aspects of performance – or nonfunctional, in the sense of architectural requirements or qualities such as safety. It’s also possible to have patterns that are subtractive or that represent the absence of something. This means that increasing the length of expressions in the pattern language does not imply a monotonic increase in the number of system components, functions, or interconnections. If the pattern language includes the means to extend the language by users of the language, then it has the capability, in principle, to encompass emergent phenomena and invention, in the sense of expanding the design space. But this is still an open research question [15]. Alexander conceived of a pattern language as a vehicle for social learning and knowledge. By utilizing the pattern language in its society, an individual designer or builder could benefit from that collective social knowledge while having sufficient scope for his own creative expression or contextual problem solving. Drawing in these concepts, the method called Design Patterns has been applied to software engineering [16]. However, the emphasis is on creating architectural templates with a relatively small number of patterns. There has been less emphasis on the language aspect – the grammar and the open-ended generative capacity. Also, this approach is limiting because it aims to provide fairly specific guidance for software engineers [15]. It does not include the concepts of capability and valuation, for example. At its current stage of development, the method of Design Patterns is not well suited to our task. Instead, we must go back to Alexander’s original ideas and add formalisms. We embrace the notion that a pattern language is a social creation and a vehicle of social knowledge made accessible to individual agents in a way that is adaptable to their circumstances. The implication is that computational models should start with an existing pattern language in which the agents are fluent and competent. It’s not advisable to expect that computation agents will create a pattern language from tabula rasa. During execution of the model, agents will make use of the pattern language in expected and (perhaps) unexpected ways to explore an emergent design space. The pattern language might adapt and shift with use but enabling full evolution of the language is not feasible with the current state of knowledge. Second, we need to formalize the definition of patterns in the context of innovation in socio-technical systems, and specifically to operate on business model elements that we have previously described, including capabilities. In general, patterns are defined by these properties [6] [16] p3: • A name that is a succinct description of a design problem and its solution • The problem that the pattern is applied to
•
The forces influencing the problem that must be balanced in any possible solution • The solution to that problem • Examples of the pattern in use • Possible consequences of using the pattern • Associated patterns, either above, below, or at the same level in the language hierarchy It is sufficient to name each pattern with a unique identifier and a token that allows designers to read diagnostic output involving the pattern language, e.g. parsing or semantic processing. The key elements to formalize are: ‘problem’, ‘forces’, and ‘solution’. Drawing on our previous formal definition of capabilities, we define the set of ‘problems’ P as:
ui ∈U : a vector of unresolved system conflicts di ⊆ U : a vector of system conflicts dependent on ui fi ∈F : a vector of possible conflicting forces in ui ui = { fi , di }
P0 ⊆ U : a set of problems facing an agent at t = t0 pk ∈P0 : a single problem involving one or more
{
}
unresolved conflicts u1 ,…,u j and dependencies In other words, problem sets are sets of unresolved system conflicts at the level of capabilities, involving one or more conflicting forces. Since there are dependencies between unresolved system conflicts, we account for these relations in a dependency graph. Thus, a problem pk for a given agent is a subset of unresolved system conflicts and their dependencies at any given time. The notion of ‘forces’ is more conceptual than in physical systems. It relates to the inner dynamics of each capability and how a set of capabilities can work together in a holistic way. Informally, a ‘force’ is any thing that shapes the capability in terms of its performance functions, cost functions, and ontological aspects –functions, behaviors, and structures. A positive ‘force’ enhances or amplifies a capability in its performance dimensions X; while a negative ‘force’ diminishes or attenuates a capability. ‘Forces’ are in conflict if they simultaneously produce both desirable and undesirable effects. Desirable and undesirable are evaluated through the valuation function V. We can formalize these notions through the contextual affordance and the FBS ontology formalisms described in section C.
We define set of ‘forces’ F as:
Finally, a ‘solution’ is a set of operations that modify the agent’s business model and capabilities. Due to space limitations we omit the formal description.
F : the set of all forces X d : the performance dimensions in F F0 ⊆ F : the set of forces for a given agent at t = t 0 fi ∈F0 : a single force FCi ⊆ F0 : the set of forces applicable to capability C j
IV. EXAMPLE: EMAIL SPAM AND ANTI-SPAM
Φ ( i ) : the force-field operator
f : Φ ( R, L,V, F , B , S ,V ) → R d m
(
)
fi : Φi C j (R, L,V, F , B , S ,V ) → R d j =1
In summary, ‘forces’ are real-valued vectors in d dimensions or fewer. The ‘force-field operator’ can be implemented as an algorithm that evaluates the gradient field in the neighborhood of a given capability, returning the force vector. In pseudo-code:
(
1 : function force-field α , β ,C j (R, L,V, F , B , S ,V ),budget 2:
constant d ← sizeOf(X )
3:
define force : R d; neighboringValuation : R d
4:
)
define neighbors : X budget ; neighborhood : {neighbors}
5:
force ← {}
6:
currentValuation ← valuation α , β ,C j (R, L,V, F , B , S ,V ), X
7:
for i ← 1 to budget do
8:
(
(
neighborsi ← perterbation α , β ,C j (R, L,V, F , B , S ,V )
)
)
9 : end for 10 : j ← 1 11 : for all x ∈ X ∧ j ≤ d do 12 :
neighboringValuation j ← valuation ( neighborhood, x )
13 :
force j ← neighboringValuation j − currentValuation
Email spam and anti-spam defenses provide a good setting to demonstrate our methods in the context of an innovation arms race [17][18], and also because the setting is relatively simple. Another benefit is that there is more empirical data available. This example is a stylized history from the period 2005-7 when image spam first appeared in volume [19]. Previously, all email spam was some combination of plain text, http links, html, or attachments. In image spam, which first appeared in volume in 2005, the entire email content consisted of one or more images that rendered as text or graphics on the consumers email client. The flow diagram in Figure 4 shows the generic structure of the adversarial system. In our example, we only consider attacker innovations. Defender capabilities are held constant, and we only consider the defensive capability of anti-spam filtering. Furthermore, we consolidate spam filters at the Internet Service Provider (ISP) and consumer into one functional entity held at the ISP. Consumers are exogenous in this simple version, and their response rate to spam is fixed. In this example, assume that the spammers are acting individually and using a simple direct marketing business model selling web site subscriptions. This includes capabilities for collecting email addresses, ‘cleaning’ the list, generating spam content, sending emails, receiving response emails, and fulfilling the responses (turning on subscriptions).
14 : j ← j +1 15 : end for 16 : return ( force) 17 : end function
This algorithm evaluates a neighborhood around a current capability, where ‘neighborhood’ is defined as capabilities that differ only by small variations. This is the same as the definition of neighborhood in fitness landscapes used in evolutionary computation. Because the number of descriptive parameters is potentially very large, this algorithm takes a sampling approach by generating a budgeted number of points in the neighborhood. Then each sampled point is evaluated by the valuation function(s) V, compared to the current valuation, across the relevant performance dimensions Xd. The difference between the neighboring valuations and the current valuation yields the force vector. Figure 4. Flow diagram of spam and anti-spam
The profit model for a spammer is given by:
π : profit per period V : volume of spam sent per period d : delivery rate (i.e. % spam that evades filter) r : consumer repsonse rate m : monetary value per response C : total costs per period (almost fixed)
π = (V ⋅ d ⋅ r ⋅ m ) − C The pattern language syntax that describes the spammer’s current business model and capabilities is given by: ← ← ← "email" ← "web site subscription" ← "text content"
Assuming that r, m, and C are fixed and can’t be influenced in the short run, the spammer can raise profits only by increasing volume V or improving the rate of delivery d, or both. Rate of delivery d is influenced by the quality of the original email address list and also by the filtering effectiveness of the ISP. In this example, assume that ISPs are using text-based filters. The spammer’s current capability generates the email message content using text only. In the text-only adversarial dynamic, spammers adjust their parameters for and in response ISPs adjust their spam filter criteria. Assume that sometime in 2005 a given spammer was no longer satisfied with the profit levels from this model, even after tuning and learning-by-doing and 100% capacity utilization on spam volume V. Let’s also say that consumer response rates r are dropping because the spammer is working so hard to evade the filter that the email content is close to unreadable. By evaluating his current capabilities, he determined that there was a system conflict in the capability. The desired result is email delivered to the consumer that the consumer will respond to it. The undesired results are that too many spam emails are getting filtered and the ones that get through are getting a poor response rate. Next, the spammer explores the space of possible innovations, focusing on alternatives for the capability. Here is a pattern available in the pattern language could be used as a fulcrum of change (described informally):
Pattern: Make your package look acceptable by appropriating the appearance a package that is already acceptable Problem: Consumers or gatekeepers reject your package because it looks like ‘junk’ Forces: The need to look acceptable; the need to promote the offer; the need to get attention; deliverability; minimize cost Solution: Find similar ‘objects’ in the consumer’s world that are accepted at face value. Then appropriate them for your purposes, preserving their external appearances. Examples: ‘official business’ envelope, parcel post package, personalized subject line, … … Using an inventive problem solving technique, the spammer could use this pattern to evaluate specific alternative capabilities that would fit. Among the alternatives is the capability to convert text to images. Because text-only spam filters ignore the content of images, this method almost completely bypasses the filter. This can be described in the following pattern language syntax: ← "text content" | ← "image capture" : "text content"
We would expect that both d and r would significantly increase, dramatically increasing spammer profit π. In a simulation, the spammer agent would discover these implications by invoking the getAffordance and evaluateAffordance functions described previously in section 2. The following is a description of the inputs, outputs and relationships of getAffordance, shown in Figure 5 for this example. The performance dimension is d, the ratio of emails delivered to emails sent. The capability being queried is ‘evade filters’ which provide the affordance ‘avoid detection’. (These names refer back to the formal ontology used in this model. Computationally, they are simply object names that encapsulate the specifics of the contextual affordance algorithms.) The spammer agent provides a list of inputs, α, that describe the specific context of this capability. This includes the set of functions E, which are used to evade filters. These include combinations of email formats (e.g. plain text, HTML, CSS, image, JavaScript, etc.) and evasion methods (e.g. bad word obfuscation, good word insertion, tokenization avoidance, etc.). The other agent inputs describe the sophistication of the capability relative to other capabilities (i.e. ‘novelty’, ‘effectiveness’, ‘automation’) or describe the learning required to master the capability (i.e. ‘accumulated experience’, ‘testing resources’) [20].
Figure 5. Example of getAffordances function used by spammer agent to explore the value of new capabilities (e.g. image spam) to avoid detection
The other input, β, describes from the defensive capability of spam filters, again specified in term of the email formats and evasion methods they cover, and their relative effectiveness for each combination. There are two outputs: the performance range function R and the learning function L, which are portrayed graphically in Figure 5. The performance range function can take one of two shapes, labeled by the shape parameter φ. Both function variations have a minimum d , representing the entry-level performance with no experience, and a
maximum d , representing the highest level of performance when the capability is fully mastered by tuning and learning-by-doing. In this example, d is simply the mean of a Gaussian probability distribution, but it could include other parameters. The first function shape applies to all combinations of filter capabilities. The second function shape applies only to specific filter capabilities, as in the case of image spam. The whole idea behind the innovation of image spam is that it introduces a format not previously covered by spam filters, or not covered well. The learning function L is a function of time and has two shapes specified by λ – inverse negative exponential and sigmoid. The learning function specifies the rate that performance will improve from minimum to maximum given a constant investment in learning. Finally, the arrows show how the inputs influence the outputs. E influences the domain of the performance range function. ‘Novelty’, ‘effectiveness’,
‘robustness’, and ‘automation’ affect the minimum and maximum of the performance range. ‘Automation’, ‘knowledge/skills’, ‘accumulated experience’, ‘testing resources’, and ‘learning investment’ affect the time from minimum performance (entry-level) to maximum performance (mastery). This approach to modeling affordances allows the system to model the evolution of social knowledge endogenously. When a capability is new little is known about its performance range and learning functions. Agents would receive only rough approximations when they queried getAffordance initially. After much experience, the performance range and learning functions would be know much better, and the outputs of getAffordance would be much more refined. It also provides a method to model social learning, since agents could acquire some or all of the information they need by communicating or collaborating with other agents rather than attempting to master innovation on their own. This will be a focus of future research. To wrap up this example, Figure 6 shows data regarding the relative success of image spam for a period of time. Only when ISPs implemented anti-image spam technology did it decline.
[4] [5]
[6] [7] [8] Figure 6. Image spam as proportion of all spam volume, June 2006 Feb. 2008 [19]
V. IMPLEMENTATION We are implementing a prototype of adversarial innovation in the context of email spam and anti-spam defenses using Mathematica. Our implementation can be viewed as a two-level game, where the base level is the daily competition between sending spam email and filtering that email (elements shown in Figure 4). This base level is simulated using a System Dynamics model in NetLogo, based on the work of Pavlov, et al. [18] who model email spam as a Tragedy of the Commons. The second level of the model is the innovation game where adversaries seek to modify or augment their capabilities to change the payoffs and options in the base-level game. Mathematica is a useful platform for this application because it has powerful symbolic computing capabilities through rewrite rules and functional programming. It also has good capabilities for computing on structures such as graphs, matrices, and lists, along with predicate logic. Though we are still in the early stages, our progress on the prototype has shown that the formal methods described in this paper are both feasible and are useful in building multi-agent system (MAS) simulations of adversarial innovation. ACKNOWLEDGEMENTS This work was supported by a grant from the National Science Foundation (#0915482) in the Directorate of Social, Behavioral, and Economic Sciences.
[9] [10] [11] [12]
[13]
[14]
[15]
[16] [17]
[18]
REFERENCES [1] [2] [3]
A.J. Robson, “Complex Evolutionary Systems and The Red Queen”, The Economic Journal, Vol. 115, No. 504, 2005 , pp. 211-346. C.B. Bingham, K.M. Eisenhardt, N.R. Furr, “What Makes a Process a Capability?”, Strategic Entrepreneurship Journal, Vol. 1, No. 1-2, 2007, pp. 27-47. [S.-C. Chang, R.-H.; Chen, H.-Y. Tsai, H.-C. Tsai, “A Conceptual Framework of the Capabilities of Knowledge Application and Innovation Capabilities”, International
[19]
[20]
Conference on Business and Information, July 11-13, 2007, Tokyo, Japan. K.M. Eisenhardt, J.A. Martin, “Dynamic Capabilities: What Are They?”, Strategic Management Journal, Vol. 21, No. 10-11, 2000, pp. 1105-1121. A. Osterwalder, “The Business Model Ontology A Proposition In A Design Science Approach”, PhD Thesis, Universite De Lausanne, Ecole Des Hautes Etudes Commerciales, 2004. Alexander, C., S. Ishikawa, M. Silverstein, A Pattern Language: Towns, Buildings, Construction. Oxford University Press, Oxford, UK, 1977. J. Wallin, “Operationalizing Competences”, Advances in Applied Business Strategy, Vol. 7, 2005, pp.151 – 179. J. Hagel III, M. Singer "Unbundling the Corporation." McKinsey Quarterly, Vol.3, 2000, p. 148-161. R. Grant, “A Resource-based Theory of the Firm”, Calif. Management Review, Vol. 33, No. 3, 1991, pp. 114-135. Davis, P.K., Analytic Architecture for Capabilities-Based Planning, Mission-System Analysis, and Transformation, RAND Corporation, Santa Monica, CA, 2004. J. S. Gero, “Design Prototypes: A Knowledge Representation Schema For Design”, AI Magazine, Vol. 11, No. 4, 1990, pp. 26-36. J. Cagan, M.I. Campbell, S. Finger, T. Tomiyama, “A Framework for Computational Design Synthesis: Model and Applications”, Journal of Computing and Information Science in Engineering, Vol. 5, 2005, pp. 171-181. S.A. Kauffman, E.D. Weinberger, “The NK Model of Rugged Fitness Landscapes and its Application to Maturation of The Immune-Response”, Journal of Theoretical Biology, Vol. 141, No. 2, 1989, pp. 211-245. M. Papasimeon, “Modelling Agent-Environment Interaction in Multi-Agent Simulations with Affordances”, Australian Department of Defense, Defense Science and Technology Organisation, DSTO–RR–0349, 2010. T. Hoverd, “Generative Patterns for Software”, Qualifying Dissertation, University of York, July 2008, http://www.cosmosresearch.org/docs/THsQualifyingDissertation.pdf, last retrieved November 1, 2010. Gamma, E, R. Helm, R. Johnson & J. Vlissides, Design Patterns: Elements of Reusable Object- Oriented Software, Addison-Wesley, Reading, MA, 1994. P.H. Calais Guerra, et al., “Exploring the Spam Arms Race to Characterize Spam Evolution”, CEAS 2010 - Seventh Collaboration, Electronic messaging, Anti-Abuse and Spam Conference, Redmond, Washington, July 13-14, 2010. O. Pavlov, N. Melville, R. Plice, “Mitigating the Tragedy of the Digital Commons: The problem of unsolicited commercial e-mail”, Communications of the Association for Information Systems, Vol. 16, 2005, pp. 73-90. Marshal, “The Rise and Fall of Image Spam”, White Paper, Marshal Ltd., March 2008, http://www.m86security.com/newsimages/trace/RiseandFal lofImageSpam_March08.pdf, last accessed Nov. 1, 2010. J. Graham-Cumming, “The Spammers' Compendium”, September 2007, http://www.jgc.org/vb2007.pdf, last accessed Dec. 21, 2010.
