Simulation-based performance evaluation of cloud applications Antonio Cuomo and Massimiliano Rak and Umberto Villano
Abstract As the cloud paradigm gains widespread adoption, performance evaluation and prediction of applications remain daunting tasks that have not yet been fully accomplished. Nevertheless, reliable performance numbers are the key to take the cloud to the next step, in which it will be possible to predict the maintenance cost of the applications and to introduce richer Service Level Agreement between the service provider and consumer. In this paper we propose a methodology based on both benchmarking and simulation which aims at predicting the performance of cloud applications developed trough the mOSAIC framework. The methodology is general enough to be of interest to other component-based cloud application development platforms which are emerging in these days.
1 Introduction The emerging Cloud Computing paradigm founds on a pay-per-use model where all the resource provisioning is delegated to the network (behind the scene, one or more remote cloud providers). Resources are acquired only when actually needed and charged on the basis of their effective usage. In addition, as described in the NIST [5] definition, resources are offered in a self-service manner, in which no human involvement is needed other than the request from the service user. As the cloud paradigm grows in maturity, upper layers of abstractions are appearing in research and enterprise cloud offerings which provide new building blocks to ease the development of cloud applications. Founding on higher level abstractions like messaging and data storage systems, these applications are designed independently from Antonio Cuomo, Umberto Villano University Of Sannio e-mail: {antonio.cuomo, villano}@unisannio.it Massimiliano Rak Second University of Naples e-mail:
[email protected]
1
2
Antonio Cuomo and Massimiliano Rak and Umberto Villano
the specific provider, and then executed on platforms deployed on the resources acquired by different cloud providers. Such applications are able in a (semi) automated way to acquire new cloud resources (i.e. components and virtual machines) as the service demand grows. The mOSAIC project [7] is one of the first examples of this approach to cloud application development. While it is well known that cloud-based services tend to exhibit good scalability as the number of user requests grows and more resources are elastically acquired, very few results are at the state of art available on the prediction and evaluation of cloud application performance. In our opinion, this is a research problem worth of further investigation, as a reliable prediction of performance in the cloud will enable a more thorough evaluation of business risks (by predicting application maintenance costs) and the introduction of richer Service Level Agreements for the negotiation of the quality of service between the provider and the consumer. In this paper we propose a simple methodology to obtain performance prediction of mOSAIC-based cloud applications. The proposed approach is based on the following idea: for a target application, it is possible to build up a set of custom benchmarks to drive a simulation model that is able to predict the performance of the application on the set of available resources. The mOSAIC benchmarking framework helps in building custom benchmarking applications, which can be run on the target resources on-the-fly. The numbers obtained in the benchmarks are used to determine the parameters of simulation models of the cloud components: these models are described through JADES, a Java-based library which allows to specify and evaluate process-oriented discrete-event simulations. The main drawback of such an approach is, obviously, that it is resourceconsuming (so it affects the overall application resource usage). On the plus side, this operation can be done at application startup or when the application is not under stress, and offers clear optimization of the resource usage. A main goal of the methodology will be to limit the resource consumption of the simulation and benchmarks execution. The remainder of the paper is structured as follows. Section 2 describes the structure of mOSAIC applications and the proposed performance evaluation methodology. The following section 3 shows how to apply the proposed methodology with a simple case study. The paper ends with a section dedicated to conclusions and plans for future work.
2 Performance evaluation of mOSAIC applications In mOSAIC, a cloud application is structured as a set of interconnected components running on resources (i.e. Virtual Machines) leased by a cloud provider. The mOSAIC Platform, which enables mOSAIC application execution, manages the resources in a transparent way. Pre-defined mOSAIC components ease the development of applications: Queuing systems (RabbitMQ and zeroMQ) are used for component communications, an HTTP gateway accepts HTTP requests and forwards
Simulation-based performance evaluation of cloud applications
3
them to application queues, and NoSQL data management systems like Key-Value (KV) stores and columnar databases are used to organize application data. The mOSAIC Java-based API enables the development of application-specific components in the form of cloudlets: these are stateless, event-driven and asynchronous components that can self-scale on the above described platform and consume any kind of cloud resource such as queues, independently of the technologies and the API they adopt, through a wrapping system [6, 1]. A mOSAIC cloud application is described as a whole in an Application Descriptor, which lists all the application components, the cloud resources (i.e. queues and key-value stores) and the details of their interconnections. Using the descriptor as a recipe for instantiating the components on the mOSAIC platform, the resulting application can then be provided in the form of Software-as-a-Service, to be accessed by final users. Moving from these premises, we propose a methodology to generate directly from the application descriptor a performance model of the application that can be used to predict its performance, to tune its execution and to form the basis for Service level Agreement negotiation. The methodology can be schematized in the following steps: (1) Derive the simulation model from the Application Descriptor, (2) Derive the benchmarking applications from the Application Descriptor, (3) Execute the benchmark and collect the results, (4) Execute the simulation using benchmark results as parameters. Simulation generation. In the first step, simulation models are derived directly from the application descriptor, by substituting the mOSAIC components with equivalent simulation models. The simulation models are organized in: the logical application, a set of simulation processes which mimic the behavior of each component in the descriptor; the resource models, that capture the behavior of execution components like the virtual machine CPUs, memory and storage subsystems; a mapping function which associates the application components with the resources on which they execute. Benchmark generation. For the second step, mOSAIC offers a benchmarking framework which helps the mOSAIC user to build up custom benchmarks targeted to specific performance goals. Through the framework, it is possible to derive dedicated benchmarking application for each of the application component in the descriptor. Benchmarks in the cloud A notable feature is that the benchmarks are themselves mOSAIC applications that can be run on the target resources. The framework includes workload generators, components which help in controlling benchmark execution and components dedicated to gather benchmarking results. Simulation in the cloud The simulation models are also mOSAIC applications. In recent work, our group proposed mJADES [2, 3], a system able to produce simulation tasks from simulation jobs and to run them in parallel by means of multiple instances of the simulation engine, which are executed as mOSAIC cloudlets. The output variables or performance indicators are successively aggregated as needed, and presented to the final user. The mJADES application uses the JADES (JAva Discrete Event Simulator) library described in [4] to produce the simulation models.
4
Antonio Cuomo and Massimiliano Rak and Umberto Villano
3 A case study: XML Document analyzer In order to show how to apply the proposed methodology we focus on a simple case study: a mOSAIC application able to analyze XML documents. Such application is really simple in its architecture, as shown in Figure 1.
Fig. 1 the XML Analyzer mOSAIC Application
In order to predict the performance of the application we need to model and benchmark its resources (Queue and Key-Value store) and components (XML analyzer and HTTPgw). An example benchmark is shown in Figure 2: the Queue component is tested by submitting a fixed number of workload messages and evaluating the average response time and resource usage of the Queue. Due to space limitations we do not report here the similar benchmarking applications for the other components.
Fig. 2 The Queue benchmark mOSAIC Application
The simulation models are shown in Listing 1. Each component is modeled by a JADES simulation process which executes a potentially infinite loop: in practice, simulation termination is defined in the simulation driver routine (not shown here for brevity’s sake), and can be linked to a specific duration or to the occurrence of an event. The components are interconnected through mailboxes, JADES components which allow for the modeling of interprocess communication. The HTTPgw component model act as an interface for mOSAIC application towards the external world. In the simulation, this component is the source of the external requests submitted to
Simulation-based performance evaluation of cloud applications
5
the application, i.e. the simulation workload. The workload is characterized through the waitTime parameter and the generateMessage function. The former is used to model the interarrival times of user requests, while the latter can be customized to define the message characteristics. The process resource usage depends on the message and is modeled through the Evaluate function described next. Listing 1 XMLAnalyzer Simulation model void HTTPgwProcess(){ while (true) { hold(random.exponential(waitTime)); Message m = generateMessage(); mboxQueueIn.send(m); mapping.getResource().consume(Evaluate(m)); }} void queueProcess(){ while(true) { Message m=mboxQueueIn.receive(); mapping.getResource().consume(Evaluate(state,m)); mboxQueueOut.send(m); }} void XMLAnalyzerProcess{ while (simulation=active) { m=mboxQueueOut.wait() mapping.getResource().consume(m); mboxKvIn.send(m); }} void KVProcess(){ while (true) { Message m = mboxKvIn.wait(); mapping.getResource().consume(Evaluate(state,m)); }}
The Queue process executes a loop in which it receives incoming messages, consumes computational resources to account for queue processing and forwards the messages on the outgoing mailbox. The next component is the XMLAnalyzer. Differently from the other components, which are natively provided by the mOSAIC framework, this one is a cloudlet developed trough the mOSAIC API, so only the mOSAIC developer knows exactly its behaviour. Nevertheless, the detail of its interconnections in the descriptor are sufficient to generate the simulation model: the cloudlet picks up messages from the queue, perform its resource-consuming elaboration and sends the results to the KV store. Finer-grained interactions (e.g. drawing two messages for each elaboration) can be supported by manually modifying the model. The missing step is the construction of a custom benchmarking application, similar to the one adopted for the Queue. The final process represents the KV component, which as shown in the code receives incoming storage messages and consumes resources when storing data. For
6
Antonio Cuomo and Massimiliano Rak and Umberto Villano
each component of the simulation model, the benchmarking results will be used to fill the parameters of the Evaluate functions, which are regressive functions constructed on the benchmark figures.
4 Conclusions and Future Works The evelopment of cloud applications is a complex task, in which resource optimization has a relevant role. In this paper we have proposed a simulation-based methodology which helps the developer to predict the performance of his application given the acquired resources and the requests of the end users. The approach here proposed founds on performance simulations tuned by benchmarking. The benchmarks are based on ad-hoc applications that can be easily replicated in order to take into account eventual variations on the resources acquired. The included case study describes the derivation of the simulation models and benchmarks for a simple application. In future work we will extend the approach in order to enable the application to self-tune itself on the basis of the end-user requests, autonomously starting both benchmarks and simulations to take decisions on the acquisition or release of the resources. In addition we aim at integrating such functionalities with the Service Level Agreement (SLA) Framework offered in mOSAIC, so that performance simulation and benchmarks can be used in order to negotiate the SLAs and predict eventual violations. Acknowledgements This research is partially supported by FP7-ICT-2009-5-256910 (mOSAIC)
References 1. Craciun, C.D., Neagul, M., Lazcanotegui, I., Rak, M., Petcu, D.: Building an Interoperability API for Sky Computing. In: The Second International Workshop on Cloud Computing Interoperability and Services, pp. 405–406. IEEE (2011) 2. Cuomo, A., Rak, M., Villano, U.: Cloud-based Concurrent Simulation at Work: Fast Performance Prediction of Parallel Programs. In: Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), 2012 21th IEEE International Workshops on (2012). To appear. 3. Cuomo, A., Rak, M., Villano, U.: mJADES: Concurrent Simulation in the Cloud. In: 2nd International Workshop on Intelligent Computing at Large Scale (2012). To appear. 4. Cuomo, A., Rak, M., Villano, U.: Process-oriented discrete-event simulation in Java with continuations:quantitative performance evaluation. In: International Conference on Simulation and Modeling Methodologies, Technologies and Applications (SIMULTECH) (2012). To appear. 5. Mell, P., Grance, T.: The NIST definition of cloud computing. NIST Special Publication 800, 145 (2011) 6. Petcu, D., Craciun, C., Rak, M.: Towards a Cross Platform Cloud API - Components for Cloud Federation. In: F. Leymann, I. Ivanov, M. van Sinderen, B. Shishkov (eds.) CLOSER, pp. 166–169. SciTePress (2011) 7. mOSAIC Project: mOSAIC: Open Source API and platform for multiple Clouds. http://www.mosaic-cloud.eu (2010)
