To appear in Proceedings of The International Conference on Dependable Systems and Networks (DSN-2001), G¨oteborg, Sweden. Copyright c 2001.
Middleware Support for Voting and Data Fusion Christopher C. Jones, David A. Karr Distributed Systems and Logistics Dept. BBN Technologies Cambridge, Massachusetts USA fccjones,
[email protected]
David E. Bakken, Zhiyuan Zhan School of Electrical Engineering and Computer Science Washington State University Pullman, Washington USA fbakken,
[email protected] Abstract
ply arrives at the voter, which must choose one and send it on to the client. In addition to this simple scenario, clients themselves may be replicated, as with a nested invocation chain, in which case the requests sent out by the client replicas must be collated into a single request to be sent to each of the server replicas. Data Fusion [41] is similar to voting, in that it involves producing one output datum from many related input data. It is different from voting in a number of key ways. For example, there are not well-defined replicas to expect inputs from, so naming and the number of inputs to expect are issues. Another difference is that the values are not necessarily expected to be the same or even close, as with actively replicated servers. Data fusion is a key technique used with distributed sensor networks [18], which are becoming increasingly prevalent, and intrusion detection systems [5], which are becoming increasingly necessary. Middleware is a layer of software above the operating system that offers a common programming abstraction across a distributed system and helps mask the different kinds of heterogeneity inherent in distributed systems [3]. Middleware systems such as CORBA [31] have become very popular in recent years largely due to these benefits. However, current middleware support for voting is extremely limited in both research projects and in commercial products and standards, and is virtually nonexistent for the more general techniques of data fusion. The voting sup-
Middleware is a class of software systems above the operating system which is becoming widely used for programming distributed systems. Voting is a fundamental operation when distributed systems involve replicated components. However, support for voting in middleware is very limited. This paper describes issues involved with supporting voting, and more general data fusion, in middleware. We describe the Voting Virtual Machine (VVM) architecture, which can be embedded in different middleware substrates such as .NET and CORBA. We also describe its companion Voting Definition Language (VDL), which allows for portable description of voting algorithms. The VVM and VDL, together with the external VVM voting manager, provide for voting transparency and adaptive voting. Finally, we describe how the simple “byte-by-byte” value comparison schemes used in other voting middleware as well as in byzantine fault tolerant multicast systems do not work in the face of the heterogeneity inherent in distributed systems.
1 Introduction Replication is a technique commonly used to increase the availability of computer-based services. Active replication is a common form of replication [11, 38]. In this scheme, the service request from a client is multicasted to all replicas in a consistent manner (atomically and totally ordered). Each replica then processes the request and sends back a reply. One of the replicas’ replies must be chosen for the client. This process of choosing one reply from many is called voting. The reply chosen by the voting process is then used by the client, e.g., it may contain data the client has requested from the service. A simple example of this is shown in Figure 1. Each re-
voted_reply
Client
reply[1]
Server[1]
relpy[N]
Server[N]
Voter
Figure 1. Voting Context
1
2 VVM Architecture
ported in middleware generally allows at best the specification of “majority” or “all,” based only on the return value of a method. These schemes implicitly assume that all methods have a return value and that no values from returned parameters (e.g., CORBA out or inout parameters) will be used. Further, with a single recent exception [30], they are undefined where the clients are replicated, and thus, multiple requests must be collated into one.
2.1 Overview We have designed and developed a prototype version of the Voting Virtual Machine (VVM), and its companion Voting Definition Language (VDL) policy language [4]. The VVM provides a powerful, well-specified, general, portable, flexible, and adaptable mechanism for voting. It can be used stand-alone to develop and test portable voting algorithms. The VVM allows voting policies to be changed at runtime, and can be integrated with presentation layers such as those for distributed objects (CORBA, SOAP, .NET, DCOM), message-oriented middleware (IBM’s MQSeries or Microsoft’s Message Queue (MSMQ)), as well as with other kinds of middleware. Indeed, to the best of our knowledge. the VVM is the first system for voting or collation which is embeddable into different middleware substrates, and VDL is the first independent and portable voting language that has been implemented. A simplified picture of the VVM architecture is shown in Figure 2, as applied to CORBA (though it could equally well apply to any other middleware system). The dashed box represents the module labeled “Voter” in Figure 1. The VVM operates as follows. Network messages (marshalled for transmission, e.g., into the CDR format used in CORBA IIOP or with Sun’s XDR) are received independently from each replica into the unmarshal module which converts the linearized messages into an array of parameters. The parameter arrays are sent to the voter core module, which selects one set based on the current voting policy, as described in the following section. Finally, the marshal module “flattens” the voted parameters into one message, which is sent to the server or client. The voter as shown here can be configured in a gateway [36]; alternatively, it can be embedded in other middleware, e.g., in a CORBA ORB, in which case its unmarshal and marshal steps will be the ORB’s ones. The voter core module accepts inputs which inform it of the current voting policy (VDL code fragment) as well as notifying it when a failure occurs. It outputs status information on the voting, which can be used by external managers to help detect performance problems, potential intrusions, or other anomalies. This status information can be passed to a Voting Status Service (not described in this paper) to support features such as intrusion detection.
Current implementations of voting in middleware are also very inflexible in dealing with floating point values in general as well as inexact voting in particular. Every middleware system of which we are aware performs voting in a “byte-by-byte” fashion on a marshalled network message rather than voting on unmarshalled application-level data. Moreover, little progress has been made toward the ability to change voting algorithms at runtime in response to changing conditions. This would be useful, for example, to provide operating points with different tradeoffs with respect to performance, fault tolerance, precision, and correctness, and switch between them when appropriate. More details on these limitations can be found in [19, 42]. The contributions of this paper are as follows:
A description of the Voting Virtual Machine (VVM), a middleware component which can be embedded in different middleware substrates and which votes on application-level data. A presentation of the Voting Definition Language (VDL), which is used with the VVM and which allows portable and reusable coding of voting algorithms. A detailed description of why current byte-by-byte implementations of voting in middleware do not work in the face of the very kinds of heterogeneity middleware is designed to mask. Additionally, we describe why value comparisons in current byzantine-fault-tolerant multicast substrates do not work for the same reasons, and discuss the architectural implications of this.
Together, the VVM and VDL help push the complexity of adding fault tolerance away from the application programmer and towards the infrastructure.
2.2 Basic Voter Functionality
The remainder of this paper is as follows. Section 2 presents the VVM and Section 3 describes VDL. Section 4 provides VDL examples for clock synchronization and a supermajority voting algorithm. Section 5 discusses implementation and other pragmatic issues regarding the VVM. Section 6 discussed related work, and Section 7 concludes this paper and discusses future work.
We now describe the basic voting functionality provided by VVM, after first providing some definitions. A ballot is a request or reply message sent by a single replica. A vote is the process of choosing (or constructing) one message from among all the ballots. 2
iiop_msg[1..N] failure notification
unmarshal param k [1..N] current voting policy & policy params; weights
voter core
Voting Manager
voted_param k Voting statistics
marshal
Voting Status Service (VSS)
voted_iiop_msg
Current conditions & QOS requirements
Alerts to subscribers
Figure 2. VVM Architecture
in the vote. These are discussed further in Section 3.2. In each of the three states, exceptions may be thrown as directed by the VDL; in which case, an exception is returned to the client that would normally receive the vote. For example, in the quorum state it may be desirable to throw an exception if the quorum is not met after a certain timeout. An exception could be thrown in the exclusion state if too many members are excluded, or in the collation state if the number of ballots which included the voted-upon value was too small. In the last two states the voter may branch back to a previous state, based on the same status conditions on which an exception may be thrown. For example, in the exclusion state, if too many ballots are excluded, the voter can branch back to the quorum state to wait for more.
unmarshal param [1..N] k
quorum exceptions
quorum
exclusion exceptions
exclusion
collation exceptions
collation Confidence
voted_param
exception
k
confidence value
2.3 Advanced Voter Functionality marshal
The voter can be specified to perform random operations at any or all of the three states to help thwart an adversary intent on corrupting a vote; this is, of course, specified in VDL. For example, in the quorum state, the voter can be instructed to wait for a random percentage of the maximum ballots (size of the group sending the ballots) to arrive. Further, this number could change with each vote or could be the same for all votes the voter managed. In the exclusion state the voter can exclude a random number or percentage of the ballots, or all but a random number of the ballots; again, this percentage could potentially be reset at different time intervals. Finally, the collation state can randomly choose a value. This is reasonable in many applications where the particularly bad data have already been discarded in the exclusion state. Another way in which the VVM can resist corruption is by weighted voting, or giving different replicas varying amounts of trust in different states of voting. For example, in the quorum state, instead of waiting for a given number
Figure 3. States in the Voter Core
The voter core shown in Figure 2 processes a single vote through the three states, as depicted in Figure 2.2. At each stage, the VDL code for the current voting policy dictates the action of the voter core at each stage. The voter begins each vote in the quorum state, where it waits for enough ballots to arrive to begin the actual voting process. After sufficient ballots have arrived, the exclusion state is entered. Here a number of ballots can be excluded from further consideration to enable the application to tolerate value failures, e.g., those caused by a faulty (and undetected) floating point processor or a breach in security. The collation state follows exclusion. Here, one value is chosen from those not excluded. After the collation state is finished, the voter may also return a confidence value, indicating the level of confidence 3
policy name (parameters) f
of ballots, it can wait for some number of points, with each replica assigned a different number of points. When a ballot arrives, that ballot’s arrival counts towards the number of points dictated by its quorum weighting. In the collation state, operations can be performed after the remaining ballots are expanded based on a weighting. For example, suppose there are ballots with values f2,3,4g from replicas 1–3, respectively, and the weighting is f1,2,2g. After the expansion, the ballots will be f2,3,3,4,4g, and this is what a collation operation such as median or random will utilize. The weights used can be computed based on many factors, including intrusion detection feeds, randomizations, and other factors. They are set by an external manager, not specified in the VDL.
quorum (quorum op (parameters)) [ throw ex name if (condition) ] [ exclusion [label] (exclusion op (parameters)) [ replace by meanjmedianjvalue] [ [ throw exception name if (condition) ] j [ goto quorum [ (using (quorum op (parameters)) morejtotal) ] if (condition) ] ] ]+
2.4 Failure Model
g
The VVM assumes a crash failure model for the hosts it runs on. This is a reasonable assumption in many situations, because, in most configurations, the VVM will be running in the same machine or even process as the client. Thus, for all practical purposes, the client and its voter will fail together or not at all; and the client will not have to deal with the possibility of a failed voter. The VVM also assumes a Byzantine failure model for applications running on servers, but only a crash failure model for the middleware and operating system code running on them. This model can be useful for applications that use data fusion or N-version programming, where the data to be collated are generated in multiple different ways and some may be inaccurate or erroneous. If this were not realistic for a given system, then a stronger failure model (weaker assumptions) could be provided with techniques such as the network attachment controller (NAC) in the Delta-4 system [11] or with more expensive protocols similar to those in [35] or [9]. This is being pursued by Trusted Information Systems, as discussed in Section 7.1.
collation (collation op (parameters)) [ [ throw exception name if (condition) ] j [ goto quorum [ (using (quorum op (parameters)) morejtotal) ] if (condition) ] j [ goto exclusion [label] if (condition) ] ] [confidence (confidence expression) ]
Figure 4. VDL Syntax
VDL’s syntax. Square brackets (“]”), asterisks, and pluses are used to denote portions of syntax that may occur zero, one, or more times, in the traditional use of these symbols. A VDL policy begins with the policy’s name and parameters, which can include long, double, the built-in enumerated types for operations in each state. Following is the declaration of the quorum operation and its parameters. It can optionally be followed by a statement indicating that a named exception should be thrown to the client under certain conditions. These conditions are expressions built with the policy’s parameters and also condition variables that the voter core provides regarding the status of a vote. These are listed in Table 1. For example, elapsed time denotes the milliseconds since the voter was initialized for a given vote, for example at the time a client’s request was sent out. It is useful, for example, to set a timeout in the quorum state, or for confidence values. Note that in the descriptions of the confidence variables all percentages are based on the total number of ballots. The statement specifying the exclusion operation and its parameters follows the quorum statement. It can optionally be followed by a clause which indicates that, instead of being discarded, all values excluded by this operation are to be replaced by either the median value, the mean value, or a given value. The exclusion statement may contain multiple goto and exception statements in any order. A goto statement sends the vote back to the quorum state under certain conditions, which are specified with expressions involving the exact same set of parameters and variables as the throw statement for exceptions. This goto statement may also specify, with the using clause, a new operation and new parameters to use in the quorum state. This clause also specifies whether this quorum applies to all ballots received (qualifier total) or only those received after the exe-
3 Voting Definition Language In this section we describe VDL. We first define its syntax, then describe the primitive operations supported in the language. We then describe confidence values. We note that this research is not a theoretical investigation of how to represent all voting algorithms. Rather, it is a pragmatic investigation into how to support voting in middleware, though supporting as wide as set of algorithms as possible is obviously useful.
3.1 VDL Syntax Figure 4 gives an overview of the VDL syntax. Symbols for parentheses and curly braces (e.g., “g”) are part of 4
CONDITION VARIABLES elapsed time pct same num same pct remaining total num remaining total pct remaining now num remaining now pct excluded total num excluded total pct excluded now num excluded now ballots total ballots max
MEANING
STATE quorum
the time since vote initialized (msec) the percentage of same ballots the number of same ballots the percentage of remaining ballots the total number of remaining ballots the percentage of remaining ballots (after this exclusion) the number of remaining ballots (after this exclusion) the percentage of excluded ballots the total number of excluded ballots the percentage of excluded ballots (after this exclusion) the number of excluded ballots (after this exclusion) total # of ballots received, equals (num excluded total + num remaining total) total # of ballots possible, e.g. number of replicas; if ballots total, quorum does not wait for all ballots to arrive.
exclusion
>
Table 1. VDL Condition Variables
PRIMITIVES until all but random
nearest
cution of the goto (qualifier more). Note that multiple exclusion statements are allowed in a policy. If there are more than one, any exclusion that can be branched to by a goto statement must specify a label to enable branching without ambiguity. The VDL policy finally specifies the operation to be used for the collation state (there are presently no parameters to these — they just choose one), as well as the optional exceptions and branching to quorum exactly as exclusion allowed. After collation, there may be a branch to exclusion. Finally, a confidence expression may optionally be returned to indicate how much trust should be put in the vote. If an exception is thrown by the VVM, the exception type is one defined by the middleware system (middleware typically requires clients to catch exceptions, e.g., SYSTEM EXCEPTION for CORBA) so as not to introduce new exceptions into the interface. Table 2 shows the primitives for each state. Primitives in the exclusion state are applied after the values have been sorted.
collation
MEANING
k[%] wait until k[%] ballots arrive k[%] wait until all but k[%] ballots arrive n[%] m[%] wait until a random number (ranged from n[%] to m[%]) of ballots arrive lowest n[%] exclude the lowest n[%] values highest n[%] exclude the highest n[%] values furthest n[%] exclude the furthest n[%] values from the median distance e from exclude all values not within e dis[meanjmedianjvalue] tance from the position or value specified outside sigma x exclude all values more than x standard deviations from the mean distance neighbor d exclude all values that are not within distance d of their neighbor distance cluster d exclude all values that are not within [meanjmedian] d distance of their neighbor, starting at position specified cluster support d p See example in Section 4.2 inner k exclude the median plus k 1 adjak n[%]
random none median mean mean neighbor mode random
cent values, spaced as evenly as possible on each side of the median exclude the median and values nearest it exclude random exclude none choose the median value choose the mean value choose the closest value in a ballot from the mean value choose the most common value choose a random ballot’s value
n[%]
k 1
Table 2. VDL Primitives
allow pieces of code (confidence policies) to be written by a different person from the VDL programmer, and they also could potentially be reused with different VDL policies. No matter how its specified, the confidence value will allow the client to adapt with a much better granularity than the current boolean output behavior. We are also investigating utilizing the confidence value for application-level adaptation while still providing voting and replication transparency. This is enabled by using delegates, which are proxies interposed between the application client and the middleware proxy (stub), that provide the same API as that proxy (the API of which the client has programmed to). In particular, we are investigating the use of delegates from the Quality Objects (QuO) framework [43]. QuO delegates are generated based on QuO’s Structure Description Language (SDL) [24] plus the middleware’s interface definition language (IDL) and the QuO’s contract description language (CDL). The SDL allows specification of adaptation strategies which are above the middleware layer yet below (and generally transparent to) the client. Note that the voter cannot return a confidence in the method’s return value, because the method’s definition has fixed its signature. Alternate means of returning the value are being developed, analogous to a Unix system call “returning” an error report “on the side” via the errno variable. In CORBA, for example, one possibility for passing the confidence value involves context variables.
3.2 Confidence Values All other voters we are aware of allow only binary output behavior: either the voter outputs a voted value, or either fails or refuses (and perhaps throws an exception). However, the choices of throwing an exception and returning a value represent two extremes. VDL allows the expression of confidence values, with multiple uses which we are exploring, in order to provide some middle ground between these extremes. One use is to allow the client to decide how to use a reply based on how good it is perceived. The confidence value is presently specified in VDL, as described in Section 3.1. Alternately, we are investigating allowing its specification via a separate confidence definition language. This would 5
policy CNVA (d, local value) f quorum (all) exclusion (distance (d)) replace by local value throw NOTHING LEFT if (pct excluded total = 100.0) collation (mean) confidence (pct remaining total=100:0) g
policy SuperMajority () f quorum (until(60%)) throw QUORUM TIMEOUT if (elapsed time > 1000) exclusion (distance (1%)) throw BAD VALUES if (pct excluded total > 40) goto quorum (using (until (60%)) total) if (pct remaining total < 60) collation (median) confidence (pct remaining total=100:0) g
Figure 6. CNVA Clock Synchronization
policy FCA (d, p) f quorum (all) exclusion (cluster support (d, p)) throw NOTHING LEFT if (pct excluded total = 100.0) collation (mean) confidence (pct remaining total=100:0) g
Figure 5. Supermajority Voting Algorithm
4 Examples 4.1 SuperMajority
Figure 7. FCA Clock Synchronization
Figure 5 gives an example of a supermajority voting policy implemented in VDL. The policy returns a vote if 60% of the ballots are equal, where “equal” is set to mean being within 1%. An exception is thrown if this cannot be met. Otherwise a confidence value is returned that is equal to the percent of ballots that agreed with the value voted upon (chosen in collation). The policy initially waits for 60% of the ballots to arrive. It then discards all which are not equal. At this point, if more than 40% have been excluded, then a supermajority of 60% cannot be achieved, so an exception is thrown. If this does not happen, but less than 60% remain (which must be all equal), then the policy branches back to the quorum state to wait for more. If neither the branch nor the exception occur, then the supermajority has been met, and collation selects one value, the median. Finally, the confidence value is returned. This SuperMajority example can be parameterized so that the thresholds of 60% or 1% of the operation median, etc., can be passed in, for example by a voter manager or by a client application.
Among those steps, the first three can be mapped onto the VVM’s three voter states: quorum, exclusion and collation. The last step is to employ a correction algorithm to adjust the local clock, which would be the application’s function. We demonstrate the use of VDL to express two convergence algorithms: Interactive Convergence and Fast Convergence. Figure 6 gives an example of the Interactive Convergence Algorithm (CNVA) [21] implemented in VDL. The synchronizing node first collects clock values from all other nodes, and then replaces the values that are “too far away” from local value, where “too far away” means not within a given distance d from local value. If all the values are excluded (replaced), an exception is thrown. After that, the synchronizing node calculates the corrected local clock value by choosing the mean of those manipulated values. Finally, the confidence value is computed according to how many values originally collected are excluded (pct remaining total). Figure 7 gives an example of the Fast Convergence Algorithm (FCA) [27] implemented in VDL. After collecting values from all other nodes, the synchronizing node excludes those values which receives support from less than p other values, where valuea “supports” valueb means jvaluea valuebj
