Optimising Multiple Quality Criteria of Service-Oriented Software Architectures Anne Martens
Franz Brosch
Ralf Reussner
Karlsruhe Institute of Technology Am Fasanengarten 5 Karlsruhe, Germany
FZI Karlsruhe Haid-und-Neu-Str. 10-14 Karlsruhe, Germany
Karlsruhe Institute of Technology Am Fasanengarten 5 Karlsruhe, Germany
[email protected]
[email protected]
[email protected]
ABSTRACT
1.
Quantitative prediction of quality criteria (i.e. extra-functional properties such as performance, reliability, and cost) of service-oriented architectures supports a systematic software engineering approach. However, various degrees of freedom in building a software architecture span a large, discontinuous design space. Currently, solutions with a good trade-off between multiple quality criteria have to be found manually. We propose an automated approach to search the design space by modifying the architectural models, to improve the architecture with respect to multiple quality criteria, and to find optimal architectural models. The found optimal architectural models can be used as an input for trade-off analyses and thus allow systematic engineering of high-quality software architectures. Using this approach, the design of a high-quality component-based software system is eased for the software architect and thus saves cost and effort. Our approach applies a multi-criteria genetic algorithm to software architectures modelled with the Palladio Component Model (PCM). Currently, the method supports quantitative performance and reliability prediction, but it can be extended to other quality properties such as cost as well.
Service-oriented software systems are created to fulfil certain functionality, encapsulated as services provided by service components. The architecture of such systems offers several degrees of freedom for changes that do not affect the functionality of the system, but that may affect quality criteria (i.e. extra-functional properties). For example, the decision of how to deploy components on multiple servers can affect the performance of the system without changing its functionality. System quality might also be influenced by the choice of components to provide the required services (because of multiple implementation of the same functionality with different extra-functional properties) and the choice of hardware resources to run the system on. Thus, even after the functional design of a service-oriented system is determined, software architects have to make many design decisions influencing quality criteria. So far, several approaches to quantitatively evaluate quality criteria based on models of software architectures have been introduced (see [3, 14] for surveys on performance and reliability). Using them, the effects of a design decision on a quality criterion can be studied. However, design decisions can affect multiple quality criteria. In particular they can be contradicting, i.e. they may result in an improvement of one quality criterion and a degradation of another one. Furthermore, the interrelation between different quality criteria for a specific system under study is in general unknown. Together with possibly discrete value ranges, this leads to a discontinuous design space spanned by the design decisions. For considerably large software architectures, the number of design decisions becomes huge, and many single quantitative evaluations have to be conducted before finding a satisfying and efficient design. In this paper, we present initial ideas for an automated approach to optimise multiple quality criteria of serviceoriented software systems while considering their mutual trade-offs1 . We use existing multi-criteria evolutionary techniques to search the design space spanned by degrees of freedom of the software architecture. This work provides the software architect an approach that automatically derives the available design options from given degrees of freedom, searches the design space by modifying the architectural models, and evaluates single design candidates (i.e. architectural models) for multiple quality criteria. As we consider multiple, possibly conflicting quality criteria, we search for
Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures; C.4 [Computer Systems Organization]: Performance of Systems—modeling techniques; D.2.8 [Software Engineering]: Metrics—performance measures
General Terms Design, Performance, Reliability
Keywords Optimisation, Performance, Quality Attribute Prediction, Reliability, Software Architecture
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. QUASOSS’09, August 25, 2009, Amsterdam, The Netherlands. Copyright 2009 ACM 978-1-60558-709-7/09/08 ...$10.00.
INTRODUCTION
1 Up-to-date information on the approach can be found at http://palladio-approach.net/ PerOpteryx
Pareto-optimal design candidates [11], i.e. candidates that are superior to any other candidates in at least one quality criterion each. The output of our approach is a set of Pareto-optimal solutions, which allows trade-off analysis. Currently, our approach focusses on performance and reliability for software architectures modelled with the Palladio Component Model (PCM) [4], but the idea can be applied to any other quality criterion that can be automatically and quantitatively evaluated based on a model of the software architecture. The contribution of this paper is a method to (1) automatically generate architectural design alternatives given a baseline architecture model, (ii) evaluate these alternatives for several quality criteria with existing prediction techniques, and (iii) automatically find a set of Pareto-optimal architecture models with respect to performance and reliability by applying existing optimisation techniques. This paper is organised as follows. Section 2 presents related work. Section 3 presents background information on the PCM and the used performance and reliability prediction techniques. Section 4 presents our method of evolutionary optimisation, which is demonstrated by the example in section 5. Finally, section 6 concludes and section 7 presents future work.
2.
RELATED WORK
Basics about the application of search techniques in software engineering are presented in [18]. The surveys in [3, 14] give an overview about performance and reliability prediction based on models of the software architecture. We classify related work into three categories: First, we describe two approaches that allow to evaluate multiple quality criteria and also support automated improvement by deriving new designs. The next category consists of six papers that combine the evaluation of several quality criteria in one approach. The third category contains work on the (semi)automated improvement of a single quality criterion. Grunske [16] studies the improvement of two quality criteria, namely reliability and cost, to allow well-informed tradeoff decisions. Evolutionary computing is applied to search for good solutions. However, only redundancy of components is studied as a degree of freedom to improve reliability. Our approach envisions many degrees of freedom (three of which we present in this work). In [17], the same author presents a framework to predict multiple arbitrary quality criteria. However, this work does not consider optimisation or improvement yet. McGregor et al. [23] have developed the ArchE framework. ArchE assists the software architect during the design to create architectures that meet quality requirements. It helps to create architectural models, collects requirements (in form of scenarios), collects the information needed to analyse the quality criteria for the requirements, provides the evaluation tools for modifiability or performance analysis, and suggests improvements. Compared to our work, ArchE does not search the whole design space, but advances step-wise based on rules. It does not support reliability and only features a simple performance model. The architecture model is not component based. Consequently, degrees of freedom as presented here could not be readily identified. The approach of Sharma and Trivedi [27] predicts performance and reliability with a joint analysis model. Software behaviour is modelled as a Discrete Time Markov Chain.
The approach supports sensitivity analyses, but no automated improvement. The quality models of single components are single values (e.g. a failure probability). Thus, the change of quality criteria of components (e.g. due to a changed allocation) cannot be taken into account. Cortellessa and Trubiani [8] present an approach to study the trade-off between performance and different security levels based on Generalized Stochastic Petri Nets. Bondarev et al. [6] present a framework to evaluate performance and cost for a given set of candidates and supports trade-off analysis with Pareto-curve visualisation. Galotti et al. [13] present the ATOP methodology for evaluating quality criteria for service compositions. Using a probabilistic model checker, it is determined whether quality criteria requirements hold in a specific system design. However, in all three approaches, the candidates have to be specified manually and the design space is not automatically searched. The KLAPER approach [15] is suitable for evaluating performance and reliability. Software modelled in the KLAPER kernel language can be translated to e.g. discrete time Markov processes (DTMP) for reliability analyses and to e.g. extended queueing networks (EQN) for performance analysis. No support for automated improvement is included. UML profiles for performance and reliability [24] could also be used as a common design model, from which performance models (QNs) or reliability models (Markov chains) can be derived by transformation. Several approaches have been suggested to optimise or improve performance in isolation [29, 7, 25, 6], but they do not include other quality properties. The heuristics or rules to improve performance presented in these works could be very useful to improve the process presented in this work. We have presented an earlier version of this approach which focussed on performance in [22].
3.
BACKGROUND
This section describes the optimisation process in detail. Section 3.1 describes the PCM software architecture modelling language on which our current implementation is based. The two sections 3.2 and 3.3 discuss the techniques used in this work to evaluate the quality criteria performance and reliability, respectively.
3.1
Architectural Model: Palladio
The PCM [4] is a modelling language specifically designed for performance and reliability prediction of service-oriented and component-based systems. The language allows component developers to specify the behaviour and the performance properties of components in a parametrised form. Then, software architects can combine these component specifications with specifications of component assembly, component allocation, user behaviour, and resource environment to form an architectural model. A model transformation into a discrete-event simulation of generalised queueing networks allows software architects to predict various performance metrics such as utilisation or response time distributions of the system’s services as well as inner services offered by individual components. Another model transformation into Markov chains allows reliability analysis to predict the probability of failure on demand (POFOD) metric for the system’s services [19]. The Eclipse-based PCM Bench tool
allows the specification and analysis of PCM instances2 . The explicit modelling of different influence factors on software quality in the PCM (component assembly, component allocation, user behaviour, and resource environment) allows to identify parts of the model that can be changed without influencing the functionality of the system. For example, the allocation of a component from one server to another can be changed without affecting the functionality of the system. This offers a number of degrees of freedom in the architectural models that are discussed in the following section. In this work, we apply our approach to the PCM, but any other software architecture modelling language can be used if (i) it explicitly models the architecture and thus allows to determine degrees of freedom (see next section) and (ii) quantitative evaluation techniques for quality criteria based on this software architecture modelling language are available, possibly with intermediate transformation into other software models. Thus, our work does not require that all quality properties of interest are evaluated using a single evaluation approach. We do need a common software design model that contains quality annotations. The information could be initially spread over several models as long as there exist transformations that can create a common software design model.
3.2
Performance Prediction
Based on a PCM model of the service-oriented software system, there exist multiple ways to predict performance metrics. For this work, we use two techniques to predict the system’s response time. For fast predictions providing a mean response time, we use Layered Queueing Networks (LQNs) [12]. A fully automated transformation from PCM models to LQNs exists [20]. Based on the mean response time, candidate solutions can be compared. However, for the evaluation of service level agreements (SLAs) with quantiles such as “90% of requests finish in less that 5 seconds”, we need a more detailed response time metric. Here, we can use the SimuCom discrete event simulation [4] that outputs, among other metrics, the predicted response time distribution and thus supports to evaluate service level agreements with quantiles.
3.3
Reliability Prediction
Reliability prediction uses the Palladio reliability solver, as presented in [19]. It is based on software failure probabilities component developers provide as part of the behavioural specifications of their components. The method calculates the probability that a system usage scenario (that is, a sequence of system service calls issued by a system user) executes without failure. For the prediction, a fully specified PCM instance is transformed into an absorbing discretetime Markov chain (DTMC), that represents all possible execution paths through the architecture, together with their probabilities. Three special states in the chain mark the beginning, failure and success of the execution. Applying Markov theory [28] on the DTMC allows for calculation of the probability that the success state is reached from the execution start (and not the failure state). 2 The tool and screenshots are available at http://palladioapproach.net
4.
EVOLUTIONARY ARCHITECTURE OPTIMISATION
Section 4.1 describes the degrees of freedom we identify in a software architecture and that span the design space. Section 4.2 describes the applied evolutionary optimisation process. We discuss trade-off analysis in section 4.3 and the current assumptions and limitations in section 4.4.
4.1
Degrees of Freedom Influencing Quality
Service-oriented software architectures contain degrees of freedom that can change quality criteria without affecting the functionality of the system. In the following, we describe the degrees of freedom exploited in our current implementation and their effects on performance and reliability. More degrees of freedom are possible, as discussed in section 7. We consider three degrees of freedom of software architectures so far: Allocation: The allocation of service-providing components to servers can be changed. This affects both performance and reliability. If one server is less reliable, it is advantageous for reliability to allocate more components on the more reliable server. Additionally, the resulting less remote communication leads to fewer network-caused failures. On the other hand, performance may degrade if too many components are deployed on a single server and cause a high utilisation of that server. Alternative components: Software components can be replaced by other components that provide functionally equivalent services. These alternative components, if available, can have better performance characteristics or better reliability characteristics. Often, there might be a trade-off: A components with more internal fault tolerance mechanisms might have worse performance characteristics due to higher computation demands of plausibility checks. Hardware: The used hardware has an influence on both performance and reliability, too. Faster resources can make the system perform better. Reliable resources can make the system more reliable. There is a tradeoff on the resource level between speed, reliability and cost of devices. As we assume a fixed cost of devices for simplicity (see section 7), a trade-off between fast or reliable resources remains.
4.2
Evolutionary Optimisation
With multiple, possibly conflicting quality criteria such as performance and reliability, no global optimum can be found for the degrees of freedom in the service-oriented software architecture in general. For example, one candidate architecture might exhibit better response time, but less reliability, while another one is highly reliable, but has worse response time. Finding a weight to combine multiple quality criteria in one goal function is often infeasible, as this would require a trade-off decision before the actual possible solutions are known. Still, candidate solutions can be evaluated for optimal tradeoffs, i.e. for Pareto-optimality [11]. A candidate architecture is Pareto-optimal, if it is superior to any candidates in at least one quality criterion. More formally: Let a be a candidate solution, let C be the set of all candidate
Initial candidate
Degrees of freedom
3. Run search 1. Analyse design options Initial candidate
Design options
Population of candidates
3a Evolve candidates Allocation
HW proc rates
Component choice
2. Set up search 3b Evaluate each candidate Performance
Reliability
3. Run search 3c Select next generation Resulting candidates
Quality results Stop?
4. Present results
Figure 1: Evolutionary optimisation process model solutions found, and let qi , i ∈ N be the quality criterion of interest with a value domain Dqi , an evaluation function fqi : C → Dqi and an order ≤qi on Dqi so that c1 ≤qi c2 means that c1 is better than or equal to c2 with respect to qi . Then, candidate solution a is Pareto-optimal, iff ∀b ∃i ∈ N : fqi (a) ≤qi fqi (b)
derive a value domain from the degree of freedom. For example, if three different components a, b, c provide the same service s, they can be exchanged. This design option to choose one of the components to provide service s has a value domain {a, b, c}. 2. Next, the problem is translated into a general form that can be searched by metaheuristic techniques. In our case, with evolutionary computation, the genome of the candidates is defined. We use an array of double values for the encoding. Each design option is encoded in a single gene, i.e. in a place in the double array. For example, a system with the design options (“CPU processing speed server 1”, “CPU processing speed server 2”, “Component to provide service s”), could have an initial choice of (1.2GHz, 2.0GHz, b) and thus a genome of or (1.2, 2.0, 1). The value in that place in the array thus represents the choice made for that design option. In the example, the first gene specifies the CPU processing speed of server 1 to be 1.2 GHz. 3. Metaheuristic search techniques search in several iterations until a stop criterion is fulfilled. In each iteration, a population of candidates is evolved, evaluated, and some are selected for the next generation. The search starts with a population of 1: the initial candidate specified by the user. The steps of each iteration are shown on the right side of figure 1 and explained in the following.
If a candidate solution is not Pareto-optimal, then it is Pareto-dominated by at least one other candidate solution that is better in all quality criteria. The design-space consists of numerous, but countable number of candidates. The value domains of the design options are discrete. Thus, for the optimisation, we face a combinatorial optimisation problem [5, Def. 1.1]. The functions for evaluating quality criteria can be arbitrary. For example, for complex performance evaluations, only approximate numeric solutions can be found. Thus, we cannot apply classic techniques such as Branch-And-Bound [9]. Metaheuristics have been successfully applied to similar problems in software engineering [18]. In this work, we use evolutionary computation, as for example described in [5, p.284], but other metaheuristics could be used as well. Figure 1 shows the process model of our method. The method is described here exemplary for our current realisation with the PCM and a genetic algorithm, but can as well be used for other software architecture modelling languages and other population-based metaheuristic search techniques. The process starts with an initial model of a componentbased software architecture (initial candidate) and modifies it along the given degrees of freedom.
(a) Evolve: Based on the currently available candidates in the population, new candidate solutions are derived by “mutation” or “cross-over” or they are randomly created. With mutation, one or several design options are varied. For example, based on a candidate that allocates components A-C on server 1 and D on server 2, a new candidate with changed allocation could be derived that allocates A and B on server 1 and C and D on server 2. Similarly, a candidate can be derived by replacing a component by another one that provides functionally equivalent service. Which design option is changed can be decided randomly or by applying heuristics to guide the search. For example, a heuristic evolution is to move a component from an over-utilised server to a lightly utilised server. With cross-over, the design decisions of two candidate solutions are merged into one. For example, the first half of design decision values of candidate 1 and the second half of design decision values of candidate 2 are merged into a new candidate 3. In addition, candidates can be randomly created based on value range of the design option.
1. First, we analyse which design options are available in the given software model based on the given degrees of freedom. For example, if a software model contains four servers with one CPU each, we get four design options to change a processing speed of a CPU, one for each concrete CPU in the model. On the other hand, if a software model does not include several components that provide the same services, we cannot use alternative components as described in section 4.1 in this model. For the design options, we also need to
(b) Evaluate: In the second step, each newly derived candidate is evaluated for all quality criteria of interest. In our case, performance and reliability metrics are predicted as described in sections 3.2 and 3.3. As the software model contains all required annotations, the evaluation, as well as all other steps of the search, can be completely automated. (c) Select: After the evolution phase, the population has grown. In the selection phase, the popu-
response time of the system's service in seconds
6 5 4 3
All results Pareto optimal results
2
Initial candidate
1 0
0
0.02
0.04
0.06
0.08
0.1
failure probability of the system's service
Figure 2: Pareto-set
Exemplary resulting candidates with
lation is again reduced by removing less promising candidates. Thus, “fit” candidates survive for “reproduction”, whereas “lame” candidates “die”. Several strategies to select candidates for the next iteration exist. Currently, we filter Pareto-dominated candidates and only keep Pareto-optimal ones. Possible future strategies could also keep a number of weaker candidates to keep a diversified “gene pool” and avoid over-specialisation in local optima. Over several iterations, the combination of evolution and selection lets the population converge towards Pareto-optimal solutions. If the search also keeps a good diversity of candidates, we can hope to find a solution that is near to the global optimum. Several possible stop criteria for terminating the search exist. Currently, we use a predefined number of iterations. More sophisticated stop criteria could use convergence detection and stop when the global optimum is probably reached. 4. Finally, the resulting set of Pareto-optimal solutions is presented to the software architect, as exemplary shown in figure 2. The software architect can identify interesting solutions and make well-informed trade-off decisions between the multiple quality criteria.
Figure 3: The Business Reporting System modelled in the PCM ours, because the quality criteria depend on the software architecture in a non-trivial way: The software architect cannot, in general, predict the effects of design options without the help of quantitative predictions.
4.4
5. 4.3
Tradeoff analysis
Based on the results of our approach, software architects can make well-informed trade-off decisions based on the available possible candidates. They can either weight the quality criteria for importance, possibly using a decision support technique like the Analytical Hierarchical Process [26] or they can identify interesting points in the Pareto-set, e.g. where only a small reduction of reliability leads to a great improvement of performance. For example, the candidate with mean response time 2.17 seconds and probability of failure on demand 0.9911 in figure 2 is such an interesting “knee” in the Pareto-set. Note that our approach does not make the final decision itself. However, it shows the possible candidate solutions to the software architect to enable a well-informed decision. In particular, we believe that this decision is in general not possible without an approach like
Assumptions
In our approach, performance and reliability are evaluated independently from each other. This is due to two reasons: 1) Errors are infrequent, so we do not have to take them into account during performance evaluation. We only predict performance for the failure-free case. 2) We assume that the utilisation of resources and the resource demands do not affect reliability predictions. A prerequisite of our approach is that a common software model exists that has sufficient annotations to allow fullyautomated quality criteria prediction, either directly on the model or after transformation into other analysis models. The Palladio Component Model (section 3.1) allows this for performance and reliability, and we believe that it could as well be extended with other quality criteria annotations (such as costs).
EXAMPLE
In this section, we apply our approach to the so-called Business Reporting System (BRS), which is based on an industrial system. With this case study, we demonstrate the feasibility of our approach and illustrate how the current implementation works. In section 5.1, we present our example system. Then, in the following sections, we describe the application of the evolutionary optimisation process as shown in figure 1.
5.1
Business Reporting System
The BRS is a 4-tier, web-based system to monitor and manage business data. On a high abstraction level, it consists of 5 software components. Figure 3 shows how the BRS is modelled in the PCM. Clients either request business reports or specific entries from the database via the Webserver component. A Scheduler component connects the Web-
server component with the core application. The core application consists of a component ReportingEngine, which manages the creation of reports, and a component CacheInfo, which buffers data from the database for quick access. Both, the ReportingEngine and the CacheInfo query the component Database, which stores a configurable amount of entries in its tables. The Webserver, Scheduler and Database component are each allocated on a dedicated server (server 1 to 3), whereas the components ReportingEngine and CacheInfo are allocated together on server 4 (see allocation diagram in the lower part of figure 3). The model is annotated with resource demands for performance prediction and failure probabilities for reliability prediction.
5.2
Analyse Design Options
Based on the initial candidate described above, the available design options are derived based on the degrees of freedom described in section 4.1. Allocation: The allocation of the five components to the four servers can be varied. We derive five design options, one for each component: Each describes the allocation of this component and can take the value of any of the four servers. Alternative Components For each component in the system, set of available components in a repository is searched for alternatives. In our case, two alternatives can be found for the Webserver component: Webserver2 and Webserver3. Thus, one design option is created: The service processRequest can be provided by any of the components Webserver, Webserver2, or Webserver3. Hardware For the four servers, we can adjust the processing rate of the CPU resources. We assume that the processing rate affects the reliability of the processors in that the mean time to failure (MTTF) linearly decreases for a growing processing rate. As CPUs cannot be arbitrarily fast, we assume that the upper bound for CPU speed is twice the initial CPU speed.
5.3
Set up Search
The identified design options are encoded in genomes for the evolutionary optimisation. The first four genes specify the processing rate of the CPUs of the servers 1 to 4. The fifth gene specifies which component has been chosen out of Webserver (value = 1.0), Webserver2 (value = 0.0), or Webserver3 (value = 2.0). Finally, the last five genes specify the allocation of each component to the servers. One gene specifies the allocation of one component (WebServer, Scheduler, Database, ReportingEngine, CacheInfo, in that order), and can take the values 0.0 (for server 1) to 3.0 (for server 4) in discrete steps of 1. The genome of the initial candidate then is (20.0, 20.0, 26.0, 8.0, 1.0, 0.0, 1.0, 3.0, 2.0, 2.0).
5.4
Run Search
We run the search for 4 iterations and generate 5 new candidates in each iteration by crossover and mutation. For performance evaluation, we use the LQN solver as described in section 3.2. For reliability evaluation, we use the Palladio reliability solver, as described in section 3.3. We use an implementation of a genetic algorithm of the Opt4J framework [21]. The overall search took 21 seconds.
5.4.1
Evolve
Candidates are evolved by mutation and crossover. In addition, we created random candidates with design option values within the bounds described in section 5.3. Random creation: At the beginning of the search, the population has been increased by random generation of candidates. For example, a candidate no. 1 (10.01, 21.64, 44.93, 5.37, 0.0, 2.0, 1.0, 1.0, 2.0, 3.0) has been created randomly. Mutation: In a mutation step, all genes are varied. For example, a candidate that uses the Webserver component is mutated and its offspring now uses the Webserver3 component. Processing rates are mutated using Gaussian mutation of Opt4J [21]. The mutation favours small changes to large ones. For example, the processing rates (27.31, 29.09, 13, 7.58) are mutated to become (29.52, 31.28, 21.38, 11.62). Crossover: In a crossover step, a new candidate is created based on two “parent” candidates. Each gene determined by merging the double values of that gene of parent 1 and parent 2 based on a probabilistic weight. We used the implementation for crossover of the Opt4J framework which implements the operator as described in [10]. For example, the parents no. 2 (31.86, 23.68, 51.20, 9.88, 2.0, 3.0, 3.0, 3.0, 3.0, 0.0) and no. 3 (12.52, 33.66, 37.92, 12.37, 2.0, 2.0, 3.0, 3.0, 3.0, 1.0) can be used for a crossover and may produce the offspring no. 4 (12.25, 23.87, 37.87, 9.93, 2.0, 2.02, 3.0, 3.0, 3.0, 1.0) and no. 5 (32.14, 33.47, 51.26, 12.33, 2.0, 2.0, 3.0, 3.0, 3.0, 0.0). If the values for component selection and allocation become non-integers, they are rounded off to integers.
5.4.2
Evaluate
The new candidates are evaluated for mean response time (MRT) and probability of failure on demand (POFOD) . For example, the candidates mentioned above are evaluated as shown in table 1. For an easier reading, we have decoded the latter six genes in the table.
5.4.3
Select
After each iteration, Pareto-dominated solutions are removed from the population. For example, both the initial candidate 0 and candidate 5 in table 1 are dominated by candidate 1 and thus removed.
5.5
Present results
Our search stops after 4 iterations. The evaluated candidates can be visualised in a diagram as shown in figure 4. Note that only evaluated candidates are shown in the diagram, which is not the whole design space. The diagram shows all evaluated candidates (first series, e.g. candidates No 1 - 3), the Pareto-optimal candidates (second series, e.g. No 11 and 13 as the two leftmost Pareto-optimal ones) and, for orientation, the initial candidate No 0. We see that the algorithm was able to find more reliable candidates, however, at the cost of decreased performance. The six Pareto-optimal candidates are objectively superior to all other ones (including the initial candidate). Thus, software architects only have to decide between these six candidates and make trade-off considerations. If a response time of 2
No 0 1 2 3 11 13
Candidate genome (20.0, 20.0, 26.0, 8.0, WebServer, server1, server2, server3, server4, server4) (10.01, 21.64, 44.93, 5.37, WebServer2, server4, server2, server2, server4, server3) (31.86, 23.68, 51.20, 9.88, WebServer, server3, server3, server3, server3, server1) (12.52, 33.66, 37.92, 12.37, WebServer, server4, server3, server3, server3, server2) (29.48,31.26, 43.23, 8.0, WebServer, server3, server3, server2, server2, server3) (10.01, 21.57, 38.20, 11.60, WebServer2, server3, server3, server2, server2, server3)
MRT in sec 0.17 0.18 2.85 1.07 1.10 1.51
POFOD 0.02315 0.01894 0.01302 0.01931 0.01348 0.01309
response time of the BRS reporting service in sec
Table 1: Evaluation of some candidates, the last two being Pareto-optimal 7
7.
6
Future work consists of both extensions to the optimisation itself, the addition of further quality criteria, and the improvement of the evaluation of performance and reliability. In addition to performance, reliability and cost, any other quality criterion that can be automatically and quantitatively evaluated based on the common software model could be included into the presented approach. The idea of multicriteria optimisation does not put a limit to the number of quality criteria under study, because the definition of Paretooptimality is independent of the number of criteria. However, visualisation of the results is limited and only could compare two quality criteria each in a view as shown in figure 2 or three quality criteria each in a three dimensional view. Still, for more dimensions, a combination of several two or three dimensional views could be used. We assumed so far that all architectural alternatives have the same cost. A planned extension is to add cost as a third quality dimension to the problem and annotate different elements of the software model with cost. Changing the architecture with the operations mentioned in section 4.1 would lead to different overall cost and can be included in the Pareto analysis. The possibility to introduce fault-tolerance mechanisms could be explicitly added to the software models as a degree of freedom. This would allow the algorithm to explore different levels of fault-tolerance mechanisms and the resulting trade-off for performance and reliability. More explicit degrees of freedom could include
5 4 all candidates
3
Pareto-optimal candidates
2
initial candidate
1 0 0.01
0.015
0.02
0.025
failure probability of the BRS reporting service
Figure 4: Results for the BRS system seconds is sufficient, they might want to choose the leftmost candidate (10.01, 21.57, 38.20, 11.60, WebServer2, server3, server3, server2, server2, server3) with a POFOD of 0.01309 and a MRT of 1.51 seconds. Here, all components have been deployed to just two servers, which in our models lead to a lower POFOD because hardware failures become less likely. As a consequence, the performance is worse. If reliability is less critical, and response time is more important, they might choose the rightmost Pareto-optimal candidate (29.52, 31.28, 21.38, 11.62, WebServer, server4, server4, server2, server2, server1) with a POFOD of 0.02123 and a MRT of 0.11 seconds. If wanted, one of the solutions could be an input for further analysis, e.g. to determine whether SLAs with quantile requirements on response time are met.
6.
CONCLUSIONS
This paper presents a fully automated approach to optimise service-oriented architecture models with regard to performance and reliability. The approach is an iterative, evolutionary process to find Pareto-optimal solutions. The considered degrees of freedom to vary architecture models are i) component allocation, ii) component selection and iii) speed of hardware resource. We demonstrate the approach for the Palladio Component Model. The process is extensible for different modelling notations, analysis methods, and quality criteria. The presented approach helps software architects when designing systems and enables informed trade-off decisions backed up by quantitative predictions. Thus, software architects can save effort and costs for manually exploring the design space or implementing prototypes. In addition, the process can find architecture candidates not obvious to a human. Overall, the process supports a systematic engineering approach to service-oriented software with predictable quality-of-service and helps avoiding redesign after implementation and poor designs without architectural reasoning.
FUTURE WORK
• the configuration of the software stack such as the middleware. For performance, [2] presented how to automatically include performance refinements for implementation details such as middleware configuration in the software model. Thus, the different combinations of configurations could be explored as degrees of freedom. • Replication of components or servers could be added as an explicit modelling construct. • Security mechanisms could be included as explicit model elements at the cost of detoriated performance, as suggested by [27, 1]. So far, our optimisation process does not yet take quality requirements into account. In future work, we plan to integrate requirements into our process. Requirements could be included them into the evaluation of candidates by adding a goal function (i.e. an additional quality criterion) that punishes violated requirements and possibly also candidates that are too close to requirements limits to avoid risks. Additionally, requirements could be used in the selection phase, e.g.
by removing candidates that violate requirements. Similarly to quality requirements, further constraints on the software design could be reflected in the optimisation process.
8.
REFERENCES
[1] A. A. Andrews, E. Mancebo, P. Runeson, and R. B. France. A framework for design tradeoffs. Software Quality Journal, 13(4):377–405, 2005. [2] S. Becker. Coupled Model Transformations. In WOSP ’08: Proceedings of the 7th International Workshop on Software and performance, pages 103–114, New York, NY, USA, 2008. ACM. [3] S. Becker, L. Grunske, R. Mirandola, and S. Overhage. Performance Prediction of Component-Based Systems: A Survey from an Engineering Perspective. In R. Reussner, J. Stafford, and C. Szyperski, editors, Architecting Systems with Trustworthy Components, volume 3938 of Lecture Notes in Computer Science, pages 169–192. Springer-Verlag Berlin Heidelberg, 2006. [4] S. Becker, H. Koziolek, and R. Reussner. The Palladio component model for model-driven performance prediction. J. of Systems and Software, 82:3–22, 2009. [5] C. Blum and A. Roli. Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM Computing Surveys, 35(3):268–308, 2003. [6] E. Bondarev, M. R. V. Chaudron, and E. A. de Kock. Exploring performance trade-offs of a JPEG decoder using the DeepCompass framework. In WOSP ’07: Proceedings of the 6th international workshop on Software and performance, pages 153–163, New York, NY, USA, 2007. ACM Press. [7] V. Cortellessa, P. Pierini, and D. Rossi. Integrating Software Models and Platform Models for Performance Analysis. IEEE Trans. Software Eng, 33(6):385–401, June 2007. [8] V. Cortellessa and C. Trubiani. Towards a library of composable models to estimate the performance of security solutions. In A. Avritzer, E. J. Weyuker, and C. M. Woodside, editors, WOSP, pages 145–156. ACM, 2008. [9] R. Dakin. A tree search algorithm for mixed integer programming problems. Computer Journal, 8:250–255, 1965. [10] K. Deb and R. B. Agrawal. Simulated binary crossover for continuous search space. Complex Systems, 9:115–148, 1995. [11] M. Ehrgott. Multicriteria Optimization. Springer-Verlag, New York, USA, 2005. [12] G. Franks, T. Omari, C. M. Woodside, O. Das, and S. Derisavi. Enhanced modeling and solution of layered queueing networks. IEEE Trans. on Software Eng, 35(2):148–161, 2009. [13] S. Gallotti, C. Ghezzi, R. Mirandola, and G. Tamburrelli. Quality prediction of service compositions through probabilistic model checking. In S. Becker, F. Plasil, and R. Reussner, editors, QoSA, volume 5281 of Lecture Notes in Computer Science, pages 119–134. Springer, 2008. [14] S. S. Gokhale. Architecture-based software reliability analysis: Overview and limitations. IEEE Trans. on Dependable and Secure Computing, 4(1):32–40,
January-March 2007. [15] V. Grassi, R. Mirandola, and A. Sabetta. Filling the gap between design and performance/reliability models of component-based systems: A model-driven approach. Journal on Systems and Software, 80(4):528–558, 2007. [16] L. Grunske. Identifying “good” architectural design alternatives with multi-objective optimization strategies. In L. J. Osterweil, H. D. Rombach, and M. L. Soffa, editors, ICSE, pages 849–852. ACM, 2006. [17] L. Grunske. Early quality prediction of component-based systems - A generic framework. Journal of Systems and Software, 80(5):678–686, 2007. [18] M. Harman. The Current State and Future of Search Based Software Engineering. Future of Software Engineering, 2007, pages 342–357, May 23-25 2007. [19] H. Koziolek and F. Brosch. Parameter dependencies for component reliability specifications. In 6th International Workshop on Formal Engineering approaches to Software Components and Architectures (FESCA). Elsevier, 2009. [20] H. Koziolek and R. Reussner. A Model Transformation from the Palladio Component Model to Layered Queueing Networks. In Performance Evaluation: Metrics, Models and Benchmarks, SIPEW 2008, volume 5119 of Lecture Notes in Computer Science, pages 58–78. Springer-Verlag Berlin Heidelberg, 2008. [21] M. Lukasiewycz. Opt4j - the optimization framework for java. http://www.opt4j.org, 2009. [22] A. Martens and H. Koziolek. Automatic, model-based software performance improvement for component-based software designs. In 6th International Workshop on Formal Engineering approaches to Software Components and Architectures (FESCA). Elsevier, 2009. [23] J. D. McGregor, F. Bachmann, L. Bass, P. Bianco, and M. Klein. Using arche in the classroom: One experience. Technical Report CMU/SEI-2007-TN-001, Software Engineering Institute, Carnegie Mellon University, 2007. [24] Object Management Group (OMG). UML Profile for Modeling and Analysis of Real-Time and Embedded systems (MARTE) RFP (realtime/05-02-06), 2006. [25] T. Parsons and J. Murphy. Detecting performance antipatterns in component based enterprise systems. Journal of Object Technology, 7(3):55–90, Mar. 2008. [26] T. L. Saaty. The Analytic Hierarchy Process, Planning, Piority Setting, Resource Allocation. McGraw-Hill, New york, 1980. [27] V. S. Sharma and K. S. Trivedi. Quantifying software performance, reliability and security: An architecture-based approach. Journal of Systems and Software, 80(4):493–509, 2007. [28] K. S. Trivedi. Probability and Statistics with Reliability, Queuing and Computer Science Applications. Wiley & Sons, New York, NY, USA, 2nd edition, 2001. [29] J. Xu. Rule-based automatic software performance diagnosis and improvement. In WOSP ’08: Proc. of the 7th Intl. Workshop on Software and Performance, pages 1–12, New York, NY, USA, 2008. ACM.
