#
Autonomous Agents and Multi-Agent Systems, 8, 119 – 130, 2004 2004 Kluwer Academic Publishers. Manufactured in The Netherlands.
Functional Validation in Grid Computing GUOFEI JIANG
[email protected]
Institute of Security Technology Studies and Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA GEORGE CYBENKO
[email protected]
Institute of Security Technology Studies and Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA
Abstract. The development of the World Wide Web has changed the way we think about information. Information on the web is distributed, updates are made asynchronously and resources come online and go offline without centralized control. Global networking will similarly change the way we think about and perform computation. Grid computing refers to computing in a distributed networked environment where computing and data resources are located throughout a network. In order to locate these resources dynamically in a grid computation, a broker or matchmaker uses keywords and ontologies to describe and specify grid services. However, we believe that keywords and ontologies can not always be defined or interpreted precisely enough to achieve deep semantic agreement in a truly distributed, heterogeneous computing environment. To this end, we introduce the concept of functional validation. Functional validation goes beyond the symbolic level of brokering and matchmaking, to the level of validating actual functional performance of grid services. In this paper, we present the functional validation concept in grid computing, analyze the possible validation situations and apply basic machine learning theory such as PAC learning and Chernoff bounds to explore the relationship between sample size and confidence in service semantics. Keywords: grid computing, service matching, keywords and ontology, functional validation, PAC learning.
1.
Introduction
Several efforts are currently underway to build computational grids, such as Globus [1] and the DARPA CoABS Grid [2]. Grids are execution environments that enable an application to integrate geographically distributed computation and data resources. Grid computations may link tens or hundreds of these distributed resources for large-scale computations. The vision is that these grid infrastructures will connect multiple regional and national computational grids, and create a universal source of pervasive and dependable computing power that supports dramatically new classes of applications. A fundamental capability required in such a grid is a semantic broker that dynamically matches user requirements with available resources. On the web, search engines that index web pages and implement retrieval services provide this capability. Whereas humans are typically the consumers of web information, grid agents are the producers and consumers of grid resources with humans occasionally steering or interpreting the computations. When a grid computation needs a computational service such as, for example, a multidimensional Fourier transform, it discovers the required service by consulting a distributed object request broker or a matchmaking service. (An object request broker (ORB) not only locates a component or object that performs the required service but also mediates
120
JIANG AND CYBENKO
communication between the client and the service. By contrast a matchmaker only locates remote services but does not mediate communications between the client and the service. Instead, the client and the service communicate directly once their locations are made known by the matchmaking service.) Several brokering technologies and frameworks have been developed for service discovery and matching. CORBA is an infrastructure for implementing distributed applications and provides a broker as a key component [3]. Jini [4] enables the construction of dynamic and robust applications from distributed components and it also provides a lookup service. Universal Description, Discovery and Integration (UDDI) [5] registries provide a mechanism to discover and integrate web services. RETSINA [6] is an open multi-agent system that includes several semantic matchmakers.
2.
Grid services and functional validation
Remote service providers publish descriptions of their services on ORBs or matchmaking services. When a client needs a computing service, it makes a request to an ORB or matchmaking service. The request includes some keywords and possibly parameter lists for invoking the remote computing resources. The ORB or matchmaker performs a look up in its service catalog and returns candidate services with their output data structures. The client can then negotiate with these service candidates and ask for the appropriate computing services. Just like web search engines, ORB or matchmaking agents use keywords and ontologies to specify remote computing services. Deep semantics may be needed for effectively searching and matching these computing services. While ontologies specify a domain, keywords specify functionalities and properties within that domain. Several systems have been proposed for implementing such ontological matching [7, 8]. However for complicated computing services, it will be very difficult for keywords and ontologies to describe the services’ capabilities and properties precisely. For example, there are literally dozens of different algorithms for implementing Discrete Fourier Transforms (DFT). Different DFT algorithms have different inputs and outputs; they work correctly only under very specific preconditions and postconditions; they also have different computational speed and accuracy properties. Moreover, since different clients will have different requirements for services, it will be difficult to describe their various needs with standard keywords and ontologies. In fact with a large number of clients and services in a grid, it also will be a very difficult and time-consuming process to concur on a complete and standard ontology. Accordingly, it is our belief that keywords and ontologies cannot be defined and interpreted precisely enough to achieve deep semantics in a completely distributed, heterogeneous computing environment. Matching conflicts may arise between a client’s requests and a service’s responses. Some service ontologies like DARPA DAML-S [9] include a set of preconditions and postconditions for service descriptions. While this effort can deepen semantics among services, it does not go beyond the symbolic description level. Some form of functional validation of computing services is required. Functional validation is a process in which a client presents a sequence of challenges to a prospective service provider. The service provider replies to these challenges with corresponding answers. Only after the service provider’s answers are determined to be consistent enough with the client’s expectations, does the client make the actual commitment to use the service. This process is especially important for mission critical applications. While keywords and ontologies are
FUNCTIONAL VALIDATION IN GRID COMPUTING
121
Figure 1. Grid service and functional validation.
used to describe and match grid services explicitly, functional validation offers an approach to match the implicit requirements of services. Although functional validation has not yet been implemented in grid computing, the basic idea can be found in many of our daily activities. For example, we typically listen to several sample songs before we decide to buy an audio CD; we usually check the contents of a book before we decide to buy it; we use examples to test software modules during the software debugging process. Our ongoing research on agent-based systems [10] has led us to the conclusion that brokering at the purely symbolic level is not sufficient, in general, to implement truly distributed, heterogeneous multi-agent computing. We propose that two steps are required before agents commit to each other, especially in mission critical applications: service discovery and location and service functional validation. These steps are illustrated in Figure 1. Service discovery and location are performed by ORBs or matchmaking agents and are already areas of active research. However, functional validation of distributed components is a new subject of research that is essential for the success of truly heterogeneous, distributed computing grids.
3.
Possible validation situations
The fundamental question in functional validation is: Do all parties involved in the prospective computation agree on the services’ actual functionality? Our approach to
122
JIANG AND CYBENKO
answering this question is to allow the client to challenge the service provider with a sequence of test cases, x1, x2, . . . , xk. The service provider offers a sequence of corresponding answers, fR(x1), fR(x2), . . . , fR(xk) directly or indirectly. Here the client itself may or may not have independent knowledge of the correct answers, fC (x1), fC (x2), . . . , fC (xk). By validating the correctness of the service response, the client may or may not commit to using the service provided. We formalize the functional validation process as follows: Denote the client’s calling process by C and the remote service by R. C requires the evaluation of a function, fC (x), where x is the input parameter. The compatibility of inputs and outputs of services is verified by the ORBs or matchmakers so that the client always knows the interface for invoking the remote service. The remote service is expected to provide fR(x) for test samples. There are several possible situations that can arise. We will use an example of matrix inverse operations to illustrate the functional validation solutions to these situations. 3.1. C knows fC(x) and R provides fR(x) In this situation, C itself knows fC (x), the ‘‘correct’’ response, for the testing samples. C may compute fC (x) by itself or get fC (x) from another trusted or previously validated service. For example, if the client C needs a computing service for large dimension matrix inversion operations, C may choose low dimensional matrices as test cases and execute the matrix inverse operations by itself to get fC (x). Alternatively, if C has other trusted or previously validated services for matrix inverse operations, it can get fC (x) from these trusted services to validate new services. After C challenges R with these testing samples, the service results fR(x) are returned. By comparing fR(x) and fC (x), the client, C, can determine whether R is implementing the right service. 3.2. C knows fC (x) but R does not provide fR(x) In some cases, the service R may not be willing to offer the actual computing results for the test samples. For example, imagine that R is a fee-based service and is not sure whether C will use the free validation responses in its actual applications. In this case, the service R may offer hashed results g( fR(x)) for functional validation purposes. Here g is a secure hash function specified by C, R or an intermediary. For the above example of matrix inversion operations, we can use g( fR(x)) ¼ Y fR(x). Here Y is a specific singular matrix. Since Y 1 does not exist, C is not able to compute fR(x) from Y and the returned g( fR(x)). By comparing g( fC (x)) and g( fR(x)), we can formalize this situation as discussed further in 3.1. Many hash algorithms, such as MD5, are available for this approach to functional validation. An ideal hash function for functional validation would have one-to-one mapping and noninvertibility properties. (Note that g( fR(x)) ¼ Y fR(x) where Y is a singular matrix is not a one-to-one hash function so that fC (x) ¼ fR(x) cannot be deduced from g( fC (x)) ¼ g( fR(x)). This will be discussed in more detail below.) Zero Knowledge Proof theory [11] has been used to securely exchange information between corresponding parties without giving away any unnecessary knowledge. We believe the theory can also be used in this functional validation scenario.
FUNCTIONAL VALIDATION IN GRID COMPUTING
123
3.3. C does not know fC (x) but R does provide fR(x) In some applications, C may not be able to know fC (x) a priori. However C may still be able to verify R’s service in an indirect way. For example, C may request weather forecast data from a service R. C cannot verify R’s responses without a time delay. However, C can record the weather predictions and then evaluate them after the actual weather is observed. Moreover, C could try several service providers to get values of fC (x) and check the consistency of these results to verify them. Based on the temporal and spatial differences, C can verify whether R is offering the right service. In the example of matrix inverse operations, even if C does not know fC (x) ¼ x1 for the test samples, C can still verify R’s service by multiplying the input matrix x against the returned fR(x) and checking whether x fR(x) is equal to the identity matrix. 3.4. C does not know fC (x) and R does not provide fR(x) This is the most difficult validation situation. It can arise when C does not know fC (x) and the service R only provides derived or hashed responses. For some cases, we believe that this situation can be simplified to a combination of the situations described in Sections 3.2 and 3.3. One approach assumes C can find several service providers that use the same hash algorithm, in which case C can check the consistency of hashed responses to verify the services. Another approach can be illustrated by the example of matrix inverse operations. Here C does not know fC (x) and R may be only willing to offer the hashed result g( fR(x)) ¼ x1 Y as discussed in 3.2. As in Section 3.3, C can still verify R’s service by checking whether x g( fR(x)) is equal to x x1 Y ¼ Y. 3.5. Discussion We have discussed several possible situations that can arise in functional validation. While ORBs and matchmakers use ontologies to describe and match services, functional validation offers another level of verification beyond semantic descriptions prior to service commitments. However, functional validation itself is not proposed to address the trust relationship issues between clients and services. For example, the service R might modify its responses to trick the client; the service R might discontinue providing its service after the validation process is completed. To address such issue, some service rating mechanisms can be used to rank different service providers so that clients would only use services with good ratings. In this paper, we focus on service matching problems and will not address these trust relationship issues although they are extremely important. In the functional validation process, a sequence of test samples is used to challenge service providers. This leads to a fundamental question that we address next: how many test samples are needed to determine whether the service provider is offering the expected service?
4.
A general mathematical framework
In this section, we formalize the sample size question and apply some basic ideas drawn from PAC learning theory. The role of PAC learning in this context is to use as few
124
JIANG AND CYBENKO
samples as possible to determine whether or not the functionality conveyed by the results on the test samples is, with sufficiently high probability, an acceptably close approximation to the desired service functionality. If the results are deemed acceptably close enough, then the client can decide to commit to the service. In this section, we will only discuss the functional validation situation presented in Section 3.1. We assume the input space, X, is a fixed set, either finite, countably infinite, [0,1]n, or Euclidean n-dimensional space for some n 1. For the functional validation problem discussed here, we want to verify whether the service provider can offer the service requested by the client. Define a concept to be a Boolean mapping h : X ! {0,1} as follows: h(x) ¼ 1 indicates that the sample x is a positive example of f (that is, the service responds with a correct answer for the challenge x); and h(x) ¼ 0 indicates that the sample x is a negative example (that is, the service provider responds with an incorrect or unacceptable answer for the challenge x). A hypothesis class H is defined as the collection of such concepts, h, over x. The size and complexity of H of course depend significantly on the specific service being validated and the possible services that will be tested according to this procedure. The complexity and size of a concept class can be measured with the Vapnik-Chervonenkis dimension (VC dimension) [12]. We refer the reader to other sources ([12] and [13] for example) for formal definitions of VC dimension and its properties, such a discussion being outside the scope of this paper. Let P be a fixed probability distribution on X. We assume that the samples are picked independently and randomly according to the distribution P. The ‘‘concept’’ of interest is the acceptability function, F, defined as FðxÞ ¼
1 0
if jj fC ðxÞ fR ðxÞ jj ; otherwise
ð1Þ
where is an allowable, user-selectable computational error. Clearly, F(x) ¼ 1 for all x means that a service’s returned results are within the acceptable error bounds of the client’s expectations. The extent to which F differs from the constant function 1 is the extent to which the actual service’s results differ from the client’s expectations. The functional validation problem cast in these terms is reduced to the problem of determining whether the concept, as represented by F, as above, is identical to the constant function 1. The client randomly generates m pairs, consisting of samples x and the associated values F(x) computed as above from the correct function values, the function values returned by the service and the allowable error tolerance: Sm ¼ {(x1, F(x1)), (x2, F(x2)), . . . , (xm, F(xm))}. This collection of pairs is used to ‘‘learn’’ a hypothesis h 2 H about whether the service provider is offering the service required by the client. Let AC,H denotes the set of all learning functions, A : Sm ! H. By definition, A 2 AC,H is consistent if all computed acceptability function values agree with learned functions values on the samples; that is, A(Sm)(x) ¼ F(x) for all x in the sample. A(Sm) can be considered an approximation to the true acceptability function, F, and is called the learned hypothesis. Based on this PAC learned hypothesis, the client can decide whether the service provider is offering the expected service with the desired level of confidence. In particular, the closer F is to the constant function 1, the closer the service’s responses are, considered as a set overall, to the service desired by the client. Functional validation viewed in this way is a two step
FUNCTIONAL VALIDATION IN GRID COMPUTING
125
process: first we ‘‘learn’’ an approximation to the true acceptability function, F, and then we decide whether F is close enough to the constant 1 for our purposes. Now consider the problem of how many samples are needed to learn a hypothesis that is a ‘‘good enough’’ approximation to the target concept. Recall that the target concept is the true acceptability function, F, computed from the service provider’s responses and the client’s expectations. Defining the concept, C, to be this acceptability function, F, and the empirically observed and computed hypothesis of it to be h. Then following the PAC framework define errorðhÞ ¼ probx2P ½cðxÞ 6¼ hðxÞ
ð2Þ
where probx2P[.] indicates the probability with respect to the random drawing of x according to P. Mathematically we can formalize the above problem as follows: How many challenges, m, are required so that probm ferrorðhÞ "g 1
ð3Þ
where " is a user-specified accuracy parameter and is a user-specified confidence parameter. Blumer et al. [12] solved this sample size problem in the general framework of PAC learning by proving the following theorem. Theorem 1 (Blumer et al. [12]) Let H be any well-behaved hypothesis space of finite Vapnik-Chervonenkis dimension d contained in 2X, P be any probability distribution on X and the target concept, c, be any Borel set contained in X. Then for any 0 < ", < 1, given m max
4 2 8d 13 log ; log " " "
ð4Þ
independent and random examples of c drawn according to P, with probability at least 1 , every hypothesis in H that is consistent with all of these examples has error at most ". The Vapnik-Chervonenkis dimension is a combinatorial measure of concept class complexity. It assigns to each hypothesis class, H, a single number that characterizes the complexity and size of a concept class and by the above theorem, determines a bound on the sample size needed to PAC learn elements of H. Details of these definitions and the proof of the above theorem can be found in [12]. Now if we determine the Boolean concept’s VC dimension over and select an accuracy parameter " and a confidence parameter , according to the above theorem, the client can pick m samples to PAC learn a hypothesis that is probabilistically close enough to the unknown true acceptability function between the client and service functionalities. The client can decide whether the service provider is offering the right service based on whether this PAC-Learned hypothesis is close enough to the constant function 1 or not.
126
5.
JIANG AND CYBENKO
Simplified theoretical results
PAC learning theory provides a solid mathematical framework for the functional validation problem. However, the above theoretical results cannot be practically applied in grid computing easily because it is difficult to compute the VC dimension for generic problems. We simplify the complexity of the functional validation problem in this section as follows. Assume that with regard to the applications we are considering, for all elements in space X, there is a constant probability, p, that the service provider is offering the expected, that is correct, service. Under this assumption, Chernoff bounds [13] can be used to solve the sample size problem. Theorem 2 [14]: Consider independent identically distributed samples x1, x2, . . . , xm from a Bernoulli distribution with expectation p. Define the empirical estimate of p based on these samples as m P
^ p¼
xi
i¼1
ð5Þ
m
Then for any 0 < ", V< 1 if the sample size ln 0 ln 2 2"2
ð6Þ
probm ½j^ p pj " 0
ð7Þ
m then
For most critical applications, the client continues the validation process only if the service provider succeeds in offering correct responses for all previous samples. Once the client encounters a negative example, that is, the service provider responds with an incorrect answer, the client will stop the validation process with this service provider, concluding it is inappropriate, and proceed to negotiate with other service providers. With regard to this situation, the client only needs to know how many consecutive positive samples are needed for the validation process. In this case the empirically estimated pˆ should always be equal to one for the service to be acceptable to the client. Because 0 p 1 and pˆ ¼ 1, with the same sample size m defined in inequality (6), the inequality (7) can be rewritten with the following inequality: probm ½ð1 pÞ "
0 : 2
ð8Þ
Then according to Theorem 2, it is straightforward to have the following corollary if we V define ¼ : 2
FUNCTIONAL VALIDATION IN GRID COMPUTING
127
Corollary 2.1: For the functional validation problem described above, given any 0 < ", < 1, if the sample size m
ln 2"2
ð9Þ
then probm ½ð1 pÞ "
ð10Þ
In the above inequality, " is the accuracy parameter that bounds the difference between the estimated probability and the true probability, while determines the confidence in this approximation. The goal of functional validation is to get a high confidence about whether the service will provide the correct response on the next request. Given a target probability P, the client needs to know how many consecutive positive samples, m, are required to verify that the service response for next request will be correct with probability P. From the definitions of and ", it is straightforward to see that if the inequality ð1 "Þð1 Þ P
ð11Þ
is satisfied, then the service response for next sample will have probability at least P of being correct, given the Bernoulli model and other assumptions above. Note that in inequality (9), " and are both needed to compute the sample size m. After a client selects a probability P, it needs to know how to compute the sample size m under the constraint inequality (11). We can formulate this problem as the following nonlinear optimization problem: ln m ¼ min f ð"; Þ ¼ min ð12Þ "; "; 2"2 s:t:: ð1 "Þð1 Þ P and 0 < "; < 1: The above objective function has two variables. From the constraint inequality, we require 1
P : 1"
ð13Þ
P Obviously when is larger, the sample size m is smaller. If we choose ¼ 1 with 1 " P the constraint 0 < "; 1 < 1, we can transfer the above two-dimensional function 1" optimization problem to the following one-dimensional function optimization problem: 1 0 P ln 1 B 1" C C m ¼ min gð"Þ ¼ minB ð14Þ @ A " " 2"2 s:t: 0 < " < 1 P: When " approaches the limiting values 0 or 1 P, the function g(") approaches positive infinity. Unfortunately it is difficult to get a closed-form analytical solution because of the
128
JIANG AND CYBENKO
Figure 2. Sample size m vs. probability P.
logarithm in equation (14). However, it is easy and fast to compute the minimum sample size m using standard nonlinear function optimization methods such as the steepest descent method. Figure 2 shows how the sample size changes with the given probability P. Generally several factors may affect how a client chooses the probability P. For a critical application, the client may want to have a large P to get high confidence. For a trustable service provider, the client can trust the quality of its services and choose a small P to speed the validation process. Meanwhile, for a more complicated service, the client may want to have an even larger P to validate the service. There is always a tradeoff between the functional validation efforts and the confidence in validated services.
6.
Discussions and related work
Although we have only analyzed the sample size problem for the situation described in Section 3.1, the result can be easily extended to other validation situations. The only difference is how to determine whether a sample is a positive sample or not. For all of these situations, if the service response to a sample satisfies the client’s expectation, we can regard this sample as a positive sample. For example, in the situation of Section 3.1, if || fC(x) fR(x)||X , the sample x is regarded as a positive sample. In the situation of Section 3.2, a sample x can be regarded as a positive sample if the hashed results satisfy ||g( fC(x)) g( fR(x))||X . As we mentioned earlier, functional validation is proposed to offer another level of verification beyond semantic descriptions for service matching in grid computing. While keywords and ontologies are used to describe and match grid services explicitly, functional validation offers another approach to match the implicit requirements of services. In fact,
FUNCTIONAL VALIDATION IN GRID COMPUTING
129
in a short time, it is not realistic to create a large standardized ontology that will cover all the details about various grid services. It will always be difficult for large number of services to concur on a standard ontology. The results of a match or mismatch could be used as a feedback to change or extend the existing ontology. For example, if an API document doesn’t have enough details about a software library, we often use examples to check whether it meets our expectations. Functional validation can be used to serve the same purpose in grid computing. The general concept of functional validation is similar to ideas found in other areas. In software quality assurance, the black box testing method [15] tests software based on requirements and functionalities, which is similar to the functional validation concept described here. However, functional validation may have more complicated situations as described in Section 3. We are not aware of any related work or previous work on the functional validation concept as presented here or, more generally, any theory for grid service matching.
7.
Conclusions
Grid computing is an emerging technology that will fundamentally change the way we think about and use computing. As part of the grid computing infrastructure, a broker or matchmaker uses keywords and ontologies to specify grid services. However, keywords and ontologies cannot be defined and interpreted precisely enough to make service brokering or matchmaking sufficiently robust in a completely distributed, heterogeneous computing environment. This may create matching conflicts between a client’s requested functionality and a grid service’s actual functionality. In this paper, functional validation is proposed to offer another level of verification beyond semantic descriptions for service matching in grid computing. While keywords and ontologies are used to describe and match grid services explicitly, functional validation offers another approach to match the implicit requirements of services. Moreover simple machine learning theory can be applied to solve the sample size problem that arises in our functional validation approach.
Acknowledgments This work was partially supported by DARPA Grant F30602-98-0107, National Science Foundation Grant CCR-9813744 and Air Force Office of Scientific Research Grant F49620-97-1-0382.
References 1. I. Foster and C. Kesselman, The Grid Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers: San Francisco, CA, 1998. 2. http://coabs.globalinfotek.com 3. R. B. Natan, Corba-A Guide to Common Object Request Broker Architecture, McGraw-Hill: New York, 1995.
130
4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.
JIANG AND CYBENKO
W. K. Edwards, Core Jini, Prentice Hall, 1999. http://www.uddi.org http://www-2.cs.cmu.edu/fsoftagents/index.html http://logic.Stanford.edu/kif/specification.html http://www.cs.umbc.edu/kqml/ http://www.daml.org http://actcomm.dartmouth.edu and http://www.cs.dartmouth.edu/fagent/ O. Goldreich and Y. Oren, Definitions and Properties of Zero-Knowledge Proof Systems. Journal of Cryptology, vol. 7, no. 1, p. 1, 1994. A. Blumer et al., ‘‘Learnability and the Vapnik-Chervonenkis dimension,’’ Journal of the Association for Computing Machinery, vol. 36, no. 4, p. 929, 1989. M. J. Kearns and U. V. Vazirani, An Introduction to Computational Learning Theory, The MIT Press: Cambridge, Massachusetts, 1994. H. Chernoff, ‘‘A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations,’’ Annals of Mathematical Statistics, vol. 23, pp. 493 – 509, 1952. B. Beizer, Black Box Testing, John Wiley & Sons, 1995.
