Building Models of Computation of Software-as-a ...

Download PDF
3 downloads 73782 Views 1MB Size Report
Comment
application, instead of its source code, and on the monitoring of the execution of ... delivering SaaS software, whose resource consumption scales well with a ...
Building Models of Computation of Software-as-aService via Monitoring Performance Indicators Dionysis Athanasopoulos and Barbara Pernici Electronics, Information, and Bioengineering Department, Politecnico di Milano, Italy Emails: [email protected] and [email protected] Abstract—Software-as-a-Service (SaaS) is becoming a prevalent model of hosting and providing third-party software. In the era of big-data, SaaS applications tend to consume a high amount of resources. While SaaS pricing policy is usually on a pay-asyou-go basis (proportional to the resource consumption), SaaS providers do not expose the specification of the resource consumption of applications, but they only offer cost pre-calculators that are application-agnostic. Such cost pre-calculators take into account upper bounds of provided facilities (e.g., maximum size of provided data storage), instead of an estimation of the resource consumption of an application. To help SaaS providers in specifying the resource consumption of applications and in offering application-specific cost pre-calculators, we propose an initial version of an approach that captures the resource consumption. To capture it in an independent way of the technical characteristics of underlying hardware and the used measurement units, our approach builds a mathematical model of computation that estimates the computational performance of an application that is directly related to its resource consumption. The proposed approach moves beyond the state-of-the-art, since it automatically builds a model of computation based on the interface of an application, instead of its source code, and on the monitoring of the execution of application by using its executable files, instead of the monitoring or the theoretical analysis of its source code. The evaluation of our approach in a set of basic case studies shows promising results. Keywords—Service-oriented applications, software-as-a-service, model of computation, application interface, performance indicators.

I.

I NTRODUCTION

The traditional model of delivering software, which is purchased and installed on personal computers, tends to become obsolete. On the contrary, the delivery of Software-asa-Service (SaaS) is becoming a prevalent model and is based on the Service-Oriented Architecture (SOA) paradigm [1]. In SaaS, developers construct software, exposing the specification of its programming interface, without providing the source code, keeping private the possession of their software asset. SaaS providers offer their infrastructures (e.g., the cloud) for hosting the execution of such third-party software, publishing, and delivering it over the Web as a service. To deliver thirdparty software, SaaS providers usually have at their disposal only the executable files of software and its programming interface, and not its source code. Since SaaS applications are now called to serve big data requirements, they tend to consume a high amount of resources1 . In this case, based on SaaS pricing policy, which is usually 1 We hereafter use the term resources to refer to any kind of computing (e.g., CPU), storing (e.g., RAM), and transferring resources (e.g., network).

on a pay-as-you-go basis, i.e., proportional to the consumed amount of resources (e.g., Amazon policy2 ), the cost of using such applications is high. Thus, big data creates the need of delivering SaaS software, whose resource consumption scales well with a high amount of input data (e.g., RAM consumption scales linearly with the size of input data). Nevertheless SaaS pay-as-you-go pricing policy, SaaS providers do not expose neither the specification of the resource consumption nor the dependency of this consumption on application input data, but they only offer cost pre-calculators that are applicationagnostic. Such cost pre-calculators take into account upper bounds of provided facilities (e.g., maximum size of provided data storage), instead of an estimation of the resource consumption of an application depending on input data. Thus, the specification of the resource consumption, which could be used in offering application-specific cost pre-calculators, is not currently provided. To construct the specification of the resource consumption of a SaaS application in a way that is independent of the technical characteristics of underlying hardware and the used measurement units, we propose the usage of mathematical models of computation that capture the computational performance of an application that is directly related to its resource consumption. For instance, instead of specifying that RAM consumption belongs to the range of 100-500 MBytes, we propose the usage of a model of computation specifying that RAM consumption y scales linearly with the size of input data x, based on the mathematical function, y = a ∗ x + b, where a and b are constants. To automatically build such models of computation, we propose an initial version of an approach that captures the resource consumption of an application and models its computational performance. Due to the mathematical formulation of model of computation, SaaS providers can easily adopt such a model to construct a cost pre-calculator. However, the construction of such a calculator is out of the scope of the paper. Model of computation. Since SaaS providers have at their disposal the programming interface of an application, instead of its source code, we propose a model of computation as a function of input data-types of the application interface. Given that the computational performance of an application is generally affected by the values of all or a subset of its input data-types, the proposed notion of model of computation is expressed with respect to the data-types, whose values affect the application performance. We hereafter call these datatypes performance features. 2 http://aws.amazon.com

(a) The interface and the input data-types of the application.

Fig. 1.

(b) The source code of the implementation of the application.

An example of Java application: its interface, its input data-types, and the source code of its implementation.

Also, the proposed notion of model of computation is generally related to the consumption in multiple types of resources, which are hereafter called performance indicators. Finally, in the proposed notion of model of computation, the consumption, related to a performance feature in a performance indicator, is captured by a mathematical function. Overall, the proposed notion of model of computation is multi-layered, since it captures the consumption related to multiple performance features in various indicators. Building a model of computation. To build such a model, the proposed approach firstly automatically determines which of the input data-types of the application interface are performance features. In this point, our approach moves beyond the state-of-the-art approaches, since in the latter approaches performance features are pre-defined and/or are building blocks of the source code of applications. Following, our approach measures the consumption related to an examined performance feature via using a runtime monitoring mechanism. In general, such a mechanism uses either direct resource measurements (e.g., CPU cycles) or energy measurement/estimation as a key metric for assessing the resource consumption in a performance indicator. Our approach is independent of the key metric used by the underlying monitoring mechanism. In this paper, we indicatively use a monitoring mechanism, which estimates the consumed energy. In SaaS paradigm, the employment of a runtime monitoring mechanism is feasible, since SaaS providers have at their disposal the executable files of applications. In this point, our approach also moves beyond the state-of-the-art approaches, since it monitors the execution of an application based on its executable files, instead of the monitoring or the theoretical analysis of its source code, performed by the latter approaches. Finally, our approach fits monitoring performance values to proper mathematical functions via using a statistical regression method [2]. To evaluate the effectiveness of our approach, we use a set of basic case studies that cover different families of computational performance. The evaluation results are promising, since the mathematical functions of the constructed models of computations scale as expected, based on the corresponding theoretical computational performance of applications. The rest of the paper is structured as follows. Section II describes a running example, used throughout the paper for illustrating purposes, and we discuss the assumptions and the limitations of our approach. Section III specifies the notion of model of computation. Section IV describes the proposed approach. Section V evaluates the effectiveness of the approach. Section VI describes the related state-of-the-art approaches. Finally, Section VII summarizes the approach and discusses its future research directions.

II.

RUNNING E XAMPLE

Throughout the paper, we use the simple Java application, depicted in Figure 1. The programming interface ArraySorting of the application provides the single method bubbleSort, which accepts as input a list of integer arrays (Fig. 1 (a)). This method sorts each array independently of the others. The programming interface of ArraySorting, along with its input data-types Arrays and Array, are the input of our approach. Figure 1 (b) depicts the implementation of bubbleSort, which is hidden from our approach. Since, the implementation includes a loop, which iterates over the number of arrays, and two nested loops, which iterate over the size of an array, both input data-types affect the computational performance of the application. This dependency is related to the consumption in CPU indicator. It is not related to RAM indicator, since the method does not consume extra RAM (except for the amount used for representing the input data-types). Thus, our approach should determine the dependency of the application performance on its both input data-types, which are performance features with respect to CPU indicator, but not with RAM indicator. In general, as already discussed, our model includes different parts, related to multiple performance indicators. Among others, the running example illustrates the direct relation that usually exists among the input data-types, the application implementation, and the application performance. However, the computational performance of an application may also depend on further factors not present in application interface, e.g., data hidden in the implementation, used database, inter-relation between performance features. Since the current work provides an initial version of the approach, we leave as future work the consideration of these factors. III.

M ODEL OF C OMPUTATION

In a nutshell, the proposed notion of model of computation (Fig. 2) is defined through three layers. In the first layer, a model is divided into parts, one for each application method. The second layer is also divided into parts, one for each monitored performance indicator. The third layer involves the performance features of the corresponding application method. In the same layer, for each feature, the performance of the application method is modeled by a mathematical function. Our choice of using application methods and performance indicators in the first and the second layer, respectively, of model of computation is motivated by the fact that the performance of different methods are usually measured by different sets of performance indicators. For instance, the performance

Fig. 2.

The multi-layer notion of model of computation of an application.

of an application method, which updates the tuples of the tables of a database, is usually measured by using various performance indicators, such as CPU, RAM, and storage indicators. On the contrary, the performance of an application method, which sorts an array in the main memory, is usually measured by using CPU and RAM indicators. Also, our choice to use performance features in the third layer is motivated by the fact that the performance of an application method in different indicators may depend on different performance features. For instance, the performance of an application method, which sorts an array, depends on the kind of the data-type of array elements when monitoring RAM, while it depends on the array length when monitoring CPU. Prior to formally defining the notion of model of computation, we firstly specify the way in which our approach represents the programming interface of an application. Our approach does not take into account the (WSDL-based or RESTful [3]) service interface, used by SaaS providers for exposing an application as Web service, but the programming interface of an application. We chose this since such an approach should be independent of the used Web service technology3 . A. Representation of Application Programming Interface In general, the programming interface of an application consists of the signatures of the application methods. We represent it with the tuple Interface (Table I (Eq. 1)), which comprises the name of an application and the set of its methods. Next, we represent a method by the tuple Method (Table I (Eq. 2)), which consists of its name and its set of input datatypes. A data-type may be a simple (e.g., Integer), or a complex one, which consists of further data-types, organized in a tree structure. In all cases, we represent a data-type by the tuple DataType (Table I (Eq. 3)), which comprises the name of the data-type, the (possibly empty) set of its constituent data-types, and the name of a (possibly existing) grouping style. We use the field groupingStyle to represent the kind of the built-in data-structure (e.g., Java List, HashMap, etc.), used for grouping multiple instances of the same data-type (e.g., Java List). In the remainder of the paper, whenever we use the term performance feature, we refer to a data-type that affects the application performance. To illustrate the data-type definition, we return to the running example. In Figure 3, we provide the tree representation of the input data-type Arrays of bubbleSort. This 3 http://www.w3c.org/TR/ws-arch

representation includes four nodes, one for each participating data-type (code lines 1, 2, 4, and 5, respectively, in the right part of Fig. 1 (a)). For instance, given that the node dataT ype2 corresponds to the Java definition List arrays, the tuple dataT ype2 includes the value arrays for the field name, the value List for the field groupingStyle, and the value Array for the field constituents.

Fig. 3.

The tree representation of an input data-type.

B. Formal Definition of Model of Computation Based on its previous overall description, we define a model of computation of an application interface as the set M odelinterf ace (Table I (Eq. 4)). Each set member is defined as a pair of an application method and its corresponding model of computation, M odelmethod . In turn, M odelmethod (Table I (Eq. 5)) is defined as a set of pairs of a performance indicator and its corresponding model of computation, M odelindicator . Finally, M odelindicator (Table I (Eq. 6)) is defined as a set of pairs of a performance feature and its corresponding mathematical function. Such a mathematical function captures the performance of an application method, related to a performance feature, with respect to a performance indicator. We hereafter call such a function atomic model function. Inspired by the computational complexity theory, which adopts specific families of functions to model different cases of the theoretical computational performance of algorithms [4], we assume that atomic model functions belong to these families. In particular, we refer to the following families of computational performance functions: (i) constant, f = c1 ; (ii) poly-logarithmic, f = log xc1 ; (iii) linear, f = c1 ∗ x + c2 ; (iv) powerlaw, f = c1 ∗ xc2 (which subsumes the polynomial function); (v) exponential, f = cx1 . In all functions, c1 and c2 are constant factors, and x is the vector of input values of application method. We represent an atomic model function f by the tuple Y (Table I (Eq. 7)), which consists of the

TABLE I.

F ORMAL DEFINITION OF APPLICATION PROGRAMMING INTERFACE AND MODEL OF COMPUTATION .

Interf ace := M ethod :=



   name : String, methodi : M ethod

name : String,

 dataT ypei : DataT ype

(1)



(2)

   DataT ype := name : String, groupingStyle : String, constituents constituents = dataT ypei : DataT ype

M odelinterf ace := M odelmethod :=

n

n

interf ace.methodi : M ethod, modelmethodi : M odelmethod

indicatori , modelindicatori M odelindicator :=

o

 o : M odelindicator indicatori ∈ CP U, RAM, storage, . . . n

f eaturei : DataT ype, yi : Y

o

n  Y := c1 : Double, c2 : Double, kind : String | kind ∈ ’constant’, ’poly-logarithmic’, ’linear’, ’powerlaw’, ’exponential’}

values of the two constants, c1 and c2 , and one of the aforementioned five main kinds of mathematical functions. For instance, the exponential function 2x is defined based on the general form of the equation of the exponential function, y = cx1 , and the tuple in Table I (Eq. 7), as follows: y = (2, 0, ’exponential’), where c1 = 2, c2 = 0, and kind = ’exponential’. To illustrate the definition of model of computation, we return to the running example. It is well-known that the theoretical computational performance (related to CPU) of the application example depends on the cardinality of the input set of arrays and on the array length. In the tree representation of the input data-type of the application (see Fig. 3), the cardinality of the arrays set corresponds to the cardinality of the grouping style List of dataT ype2 and the array length to the cardinality of the grouping style List of dataT ype4 . In this way, dataT ype2 and dataT ype4 should be two performance features (f eature1 and f eature2 ) of the application. Based on these features, the constructed model of computation is presented in Figure 4. Observe in this figure that the constructed atomic model functions (y1 and y2 ), which capture the application performance with respect to f eature1 and f eature2 belong to the families of linear and powerlaw functions, respectively, as expected by the theoretical computational performance of the application. The construction of these functions is explained in the next section.

(3)

(4) (5) (6) (7)

with their relations, are depicted in Figure 5 and explained in Section IV-B. Finally, to illustrate the functionality of the mechanisms, we employ them in the application of the running example (Section IV-C).

Fig. 5.

The mechanisms, along with their relations, of our approach.

A. An Overview of our Approach In a nutshell, our approach firstly determines which of the input data-types of the application interface are performance features. To this end, it automatically generates proper input data to examine whether the data-types affect the application performance. However, given that an infinite number of data can be generated based on the definition of data-types, our approach generates a few representative input data that cover all cases of the application performance. To measure the consumption related to a performance feature, our approach uses a runtime monitoring mechanism that records the energy consumed by the application with respect to a performance indicator. Interestingly, our approach does not depend on specific monitoring mechanism. Finally, our approach fits recorded performance values to proper mathematical functions, via using a statistical regression method [2]. B. The Mechanisms of our Approach

Fig. 4. The model of computation of the application of the running example.

IV.

B UILDING A M ODEL OF C OMPUTATION

In this section, we firstly provide an overview of our approach (Section IV-A), which includes five mechanisms for building a model of computation. These mechanisms, along

Construction of Model of Computation. This component deals with each application method independently of the others. Specifically, the mechanism initially assumes that each input data-type of a method is a candidate performance feature for this method. The mechanism examines each performance feature independently of the others in order to reduce the high number of all possible input

data combinations. To determine whether an input datatype is a performance feature, the mechanism firstly asks from the Generation of Application Input Data mechanism to generate various instances of the examined input data-type. Following, for each generated instance, the current mechanism interacts with the Measuring Application Performance mechanism in order to measure the performance of the application method. Finally, the current mechanism interacts with the Construction of Atomic Model Function mechanism in order to fit the performance measurements in a mathematical function. If the constructed atomic model function does not belong to the family of constant functions, then the mechanism decides that the examined data-type is a performance feature. Generation of Application Input Data. It generates instances of the input of an application method via generating various instances of an examined data-type (checked whether it is a performance feature), while generating base instances for the remaining data-types of the method input. In detail, the mechanism generates instances of an examined data-type based on representative values for this data-type. To this end, the mechanism selects from the value space of the datatype a small number (e.g., 5) of representative values as follows: it divides the value space into a small number of equally sized intervals and takes the medium value for each interval. For the remaining data-types of the method input, the mechanism generates base instances, i.e., instances with the lowest possible values in data-types (e.g., arrays of unitary size). In this way, the mechanism generates a small number (e.g., 5) of cases of instances for each examined data-type. The mechanism uses the aforementioned way of selecting representative values for data-types if the latter are numerical (e.g., Integer) and do not include a grouping style. For a data-type that is not numerical and includes a grouping style (e.g., List), the mechanism selects representative values based on the cardinality of its grouping style, which essentially specifies how many times a data-type is instantiated. In any other case (e.g., String), in this version of our approach, only an instance of the corresponding data-type is generated.

estimates the coefficients in the equations of these functions by using a statistical regression method [2]. Regression selects model parameters that minimize the sum of the squares of the errors of the fit for each input value. This method is known as least-squares regression and is the typical one used in the regression analysis. Least squares approaches come into two categories: linear and non-linear. The linear least-squares method is the one that is usually used in statistical regression, since it has a closed-form solution, i.e., a model is built in a finite number of standard steps, whose number scales linearly with the cardinality of the set of performance values [2]. Our mechanism adopts linear regression and computes the mean squared error (denoted by mse) between the measured performance values and the corresponding fitted values. Based on these values, the mechanism selects the function that minimizes this error measure. C. Applying the Mechanisms in the Running Example After examining all the data-types (dataT ype1 dataT ype4 ) of the application method bubbleSort, our approach concludes that dataT ype2 and dataT ype4 are performance features. Indeed, as mentioned earlier, the theoretical computational performance of bubbleSort with respect to CPU indicator scales with the cardinality of the arrays set and the array length, which correspond to dataT ype2 and dataT ype4 , respectively. We indicatively present how dataT ype4 is determined as a performance feature. Initially, a set of five cases of representative instances (Fig. 6) of the input of bubbleSort is generated, in which base instances of all data-types are generated, except for dataT ype4 . The number of the generated instances of dataT ype4 is determined by dividing the value space of dataT ype4 into five equally sized intervals. Thus, in each one of the five overall instances of the method input, the number of instances of dataT ype4 3N 4N equals to b N5 c, b 2N 5 c, b 5 c, b 5 c, and N , respectively, where N is the maximum number of instances of dataT ype4 .

Measuring Application Performance. It invokes and monitors a method for the generated instances of its input data-types. To record the resource consumption during the method invocation, it uses the Monitoring Application Execution mechanism. To ensure the validity of the performance recordings, the current mechanism repeats the method invocation (along with its monitoring) multiple times (e.g., 10), and calculates the average of the recorded performance values. Monitoring Application Execution. It monitors the amount of the resources consumed during the invocation of an application method and measures its performance values. Recall that the performance indicators (e.g., CPU), to which the monitoring mechanism is configured, depend on the parts of the model of computation that are constructed. Construction of Atomic Model Function. To construct an atomic model function, the mechanism takes into account our assumption that the method performance is modeled by five families of functions: constant, poly-logarithmic, linear, powerlaw, and exponential functions. Thus, given a set of performance values, the mechanism constructs for each family one function that fits better to these values. The mechanism

Fig. 6.

Generated instances of the input of the bubbleSort method.

V.

C ASE S TUDIES AND M EASUREMENTS

To evaluate our approach, we implemented the research prototype, AppComMod (Application Computation Modeler). In this section, we evaluate the effectiveness of AppComMod in constructing atomic model functions that scale as expected based on the corresponding theoretical computational performance of applications. To this end, we selected a set of basic case studies that cover different cases of computational performance with respect to CPU indicator. We tuned our approach to construct only the part of the model of computation of

TABLE II.

T HE USED CASE STUDIES AND THE CONSTRUCTED ATOMIC MODEL FUNCTIONS . (a) The characteristics of the used case studies.

Application ID

Name

Method

ArraysSorting

a1

bubbleSort

ArraysMerge

a2

merge

ThreeSum

a3

count

Description set cardinality array length array1 length array2 length array length

Performance Feature Symbol Theoretical Performance w.r.t. CPU f eature1 O(f eature1 ) f eature2 O(f eature22 ) f eature1 O(f eature21 ) f eature2 O(f eature22 ) f eature O(f eature3 )

(b) The constructed atomic model functions. Application ID a1 a2 a3

Performance Feature

Atomic Model Function

f eature1 f eature2 f eature1 f eature2 f eature

y1 = (4.6 ∗ 10−5 , 0.07, ’linear’) y2 = (2.76 ∗ 10−8 , 2.11, ’powerlaw’) y1 = (8.2 ∗ 10−7 , 1.81, ’powerlaw’) y2 = (4.2 ∗ 10−7 , 1.87, ’powerlaw’) y = (5.6 ∗ 10−8 , 2.86, ’powerlaw’)

(c) Mean squared errors in the used case studies (the minimum values are in bold). f eature2 29.943945 84.996153 501.697269 2.832596 442.252171

an application with respect to CPU indicator. Concerning the monitoring mechanism, we used the (open-source) monitoring agent jalen4 [5], which measures the energy consumed by CPU (in Joule). In each case study, we used a sufficient number of instances of input data-types in order to capture the computational performance of applications. The number of the used instances differ in each case study, due to their different computational performance. The algorithmic logic of the used applications is detailed in the remainder of this section. To ease the reference to the used applications, we assign to them identifiers (a.k.a., IDs), presented in Table II (a). The results of the effectiveness evaluation for each case study are discussed as follows. Evaluation of AppComMod on a1 case study. a1 corresponds to the running example of the paper. As mentioned earlier, the theoretical computational performance of a1 with respect to CPU indicator scales linearly with the cardinality of the arrays set, and for each array, it also scales quadratically with the array length (see Table II (a)). Among all the constituent data-types of the Arrays input of a1 (dataT ype1 - dataT ype4 ), AppComMod concludes that dataT ype2 and dataT ype4 are the performance features. The results of the execution of AppComMod for input data generated based on dataT ype2 and dataT ype4 are depicted in Figure 7. Observe that we use five cases of data-type instances, whose number is uniformly distributed in the interval [5000, 25000]. The maximum number of 25000 suffices to determine the performance behavior of a1 with respect to dataT ype2 and dataT ype4 . We do not present the results for dataT ype1 and dataT ype3 , since AppComMod correctly identified that they are not performance features, i.e., modeled by constant atomic model functions. Also, observe that the performance measurements for dataT ype2 and dataT ype4 fit to a linear 4 https://github.com/adelnoureddine/jalen/tree/master/1.0

a2 f eature1 12.235906 189.328373 342.445626 0.849714 324.692638

f eature2 16.116310 157.557155 387.845744 1.498499 360.529620

a3 f eature 493.660000 270.741712 16125.193161 73.580208 26167.440000

and a quadratic atomic model function, respectively. This happens since for these two functions, as depicted in Table II (c), their mean squared error is minimum. The equations of the mathematical functions, which fit to these measurements, are given in Table II (b). f eature2

f eature1 60

1.2 CPU energy [joule]

linear exponential poly-logarithmic powerlaw constant

a1 f eature1 0.004773 0.161199 0.119637 0.005806 0.110647

CPU energy [joule]

Families

1 0.8 0.6

40

20

0.4 0 0.5

1

1.5

2

# data-type instances

2.5 ·104

Performance measurements

0.5

1

1.5

2

# data-type instances

2.5 ·104

Atomic model function

Fig. 7. AppComMod results for a1 based on its dataT ype2 and dataT ype4 .

Evaluation of AppComMod on a2 case study. a2 corresponds to an application, named ArraysMerge, and provides the single method merge. This method accepts as input two arrays of numbers and returns as output their sorted concatenation. The theoretical computational performance of merge with respect to CPU indicator is quadratic with the lengths of the given arrays (Table II (a)). AppComMod examines all the constituent input data-types of merge (see dataT ype1 -dataT ype5 in Fig. 8). Based on the results (Fig. 9), AppComMod concludes that dataT ype3 and dataT ype5 are the only performance features (f eature1 and f eature2 , respectively). We again use five cases of data-type instances, whose number is uniformly distributed in the interval [4168, 20834]. Observe that the performance measurements for both features fit to quadratic atomic model functions, whose

Fig. 8.

The input data-type of the method merge of a2 .

equations are provided in Table II (b). Our approach builds quadratic functions, since as depicted in Table II (c), their mean squared error is minimum.

CPU energy [joule]

f eature1

300

f eature2 60

200

100

CPU energy [joule]

CPU energy [joule]

0 40

20

500

40

1

1.5

Performance measurements

# data-type instances

0.5

2 ·104

Performance measurements

1

1.5

2,500

Atomic model function

# data-type instances

·104

Atomic model function

Evaluation of AppComMod on a3 case study. a3 corresponds to an application, named ThreeSum, and provides the single method count. This method accepts as input a constant number and an array of numbers, and calculates the number of the triples of the array items, whose sum equals to the given constant number. The theoretical computational performance of count with respect to CPU indicator is cubic with the array length (Table II (a)). AppComMod examines all the constituent input data-types of count (see dataT ype1 - dataT ype4 in Fig. 10). Based on the results (Fig. 11), AppComMod concludes that dataT ype3 is the only performance feature. We again use five cases of data-type instances, whose number is uniformly distributed in the interval [502, 2500]. The maximum number of used element instances is 2500, which is much lower than that used in the previous case studies. This is due to the fact that a3 has higher computational performance than a1 and a2 and hence, a lower number of instances suffices to determine its performance. Observe that the performance measurements fit to a cubic atomic model function, whose equation is provided in Table II (b). Our approach builds a cubic function, since as depicted in Table II (c), its mean squared error is minimum.

The input data-type of the method count of a3 .

AppComMod results for a3 based on its dataT ype3 .

2

Fig. 9. AppComMod results for a2 based on its dataT ype3 and dataT ype5 .

Fig. 10.

2,000

20

Fig. 11. 0.5

1,500

# data-type instances

0

0

1,000

VI.

R ELATED W ORK

Interestingly, the construction of the specification of the resource consumption of SaaS applications has not been investigated by the literature yet. From a broader perspective, our work is related with two research domains: application profilers and techniques that construct performance models. In both domains, some approaches have been proposed for procedural and some for object-oriented applications. To provide a wide perspective of the adopted approaches, we cite approaches independently of the application kind. Application profilers present information about the resources consumed in only one execution of an application. One of the earliest profiler is gprof [6], which constructs the call graph of the application source code, and estimates the portion of the execution time that was spent in each method. A more recent profiler [7], moves one step further than gprof by finding expensive execution paths of an application by exploiting the call graph of the application source code. All existing profilers differ from our approach in three main points. Firstly, they do not construct a model of computation, but they simply present information about the application performance. Secondly, they present information recorded by a single execution of an application, which is not representative of all possible input data of an application. Thirdly, they trace and examine the whole application source code that is timeconsuming for large enterprize systems. Our approach is mostly related to approaches that construct performance models. These approaches are divided into two categories: (i) those that construct models by statically analyzing the source code of applications towards constructing the theoretical complexity models; (ii) those that construct models based on multiple application executions towards capturing the application performance for any input data. To the first category belong the approaches described in [8], [9]. The drawback of the approaches of this category is that they

cannot be applied in applications of large size, since the automated calculation of theoretical computation complexity is an instance of the insolvable Halting Problem [10]. Concerning the second category, one of the earliest approaches is described in [11], according to which the average number of the execution times of an application is determined by measuring the execution times of the loops of the application source code. In another approach [12], the application performance is measured as a function of the application configuration parameters (e.g., the key width in radix sort), which however are chosen by the end-user. Another approach [13] measures the performance of applications that are executed in parallel. However, this approach does not determine performance features neither in the application source code nor in their interface. On the contrary, the performance model is constructed with respect to the number of processors on which applications run. The approach in [14] also measures the performance of parallel applications, but it constructs models by executing applications on fixed input data. The most recent approaches of the second category are described in [15] and [16]. In particular, the first approach is just a framework that aims to build accurate performance models (proposed in the literature) by executing a large number of performance tests. The second approach proposes the tool trend-prof, which constructs models of computational complexity of applications. These models predict how many times each basic block of the application source code runs, as a linear or powerlaw function of user-specified performance features. Among the approaches of the second category, our approach is mostly related to that of trend-prof. However, our approach moves beyond the approaches of the second category in the following points. Firstly, our approach automatically determines the performance features of an application, while the previous approaches accept them as input by the end-user or are manually determined by the application developers. Secondly, our approach searches for candidate performance features in the programming interface of an application, instead of source code, as happened in the previous approaches. Thirdly, our approach generates automatically instances of the application input, based on which the performance features are determined, while such instances are manually produced in the previous approaches. Fourthly, our approach records the application performance via monitoring the execution of an application based on its executable files, instead of the monitoring or the theoretical analysis of specific parts of its source code, performed by the previous approaches. Finally, our approach models the application performance by using a complete set of mathematical functions, instead of the linear and the powerlaw functions, used by trend-prof. Also, our approach automatically constructs all functions and selects the one that models in the best way the application performance, while in trend-prof, the end-user manually selects the best of the two constructed functions.

it in an independent way of the technical characteristics of underlying hardware and the used measurement units, our approach builds a mathematical model of computation that estimates the computational performance of an application that is directly related to its resource consumption. We evaluated the effectiveness of our approach on a set of basic case studies and the results were promising. A possible future direction of our work is the consideration of relations among performance features. Based on these relations, an overall function, which captures the overall performance of an application, would be constructed. Another future direction would be the investigation of further performance factors, except for the input data-types of the application interface. Another possibility in future work is the refinement of constructed models of computation based on input data, provided by actual end-users of applications. Finally, the evaluation of the effectiveness of the approach could be performed in applications of larger size. R EFERENCES [1] [2] [3]

[4] [5]

[6]

[7]

[8]

[9]

[10] [11]

[12]

[13]

[14]

[15]

VII.

C ONCLUSIONS AND F UTURE W ORK

To help SaaS providers in specifying the resource consumption of applications and in offering application-specific cost pre-calculators, we propose an initial version of an approach that captures the resource consumption. To capture

[16]

T. Erl, Service-Oriented Architecture: Concepts, Technology, and Design. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2005. M. Hazewinkel, Regression Analysis. Encyclopedia of Mathematics, Springer, 2001. R. T. Fielding, D. Software, and R. N. Taylor, “Principled design of the modern web architecture,” ACM Transactions on Internet Technology, vol. 2, pp. 115–150, 2002. C. H. Papadimitriou, Computational Complexity. Addison-Wesley, 1994. A. Noureddine, “Towards a better understanding of the energy consumption of software systems,” Ph.D. dissertation, Universit´e des Sciences et Technologie de Lille - Lille I, 2014. S. L. Graham, P. B. Kessler, and M. K. McKusick, “gprof: a call graph execution profiler,” in Symposium on Compiler Construction, 1982, pp. 120–126. G. Ammons, J. Choi, M. Gupta, and N. Swamy, “Finding and removing performance bottlenecks in large systems,” in European Conference on Object-Oriented Programming, 2004, pp. 170–194. D. L. M´etayer, “ACE: an automatic complexity evaluator,” ACM Transactions on Programming Languages and Systems, vol. 10, no. 2, pp. 248–266, 1988. M. Rosendahl, “Automatic complexity analysis,” in International Conference on Functional Programming Languages and Computer Architecture, 1989, pp. 144–156. M. Sipser, Introduction to the Theory of Computation. PWS Publishing Company, 1997. V. Sarkar, “Determining average program execution times and their variance,” in Conference on Programming Language Design and Implementation, 1989, pp. 298–312. E. A. Brewer, “High-level optimization via automated statistical modeling,” in Symposium on Principles & Practice of Parallel Programming, 1995, pp. 80–91. M. Kluge, A. Knpfer, and W. E. Nagel, “Knowledge based automatic scalability analysis and extrapolation for mpi programs,” in European Conference on Parallel Processing, 2005, pp. 176–184. R. Rugina and K. E. Schauser, “Predicting the running times of parallel programs by simulation,” in International Symposium on Parallel and Distributed Processing, 1998, pp. 654–660. D. Thakkar, A. E. Hassan, G. Hamann, and P. Flora, “A framework for measurement based performance modeling,” in International Workshop on Software and Performance, 2008, pp. 55–66. S. Goldsmith, A. Aiken, and D. S. Wilkerson, “Measuring empirical computational complexity,” in International Symposium on Foundations of Software Engineering, 2007, pp. 395–404.