Variability Driven Quality Evaluation in Software Product ... - IEEE Xplore

12th International Software Product Line Conference

Variability Driven Quality Evaluation in Software Product Lines1 Leire Etxeberria, Goiuria Sagardui Computer Science Department, University of Mondragon, Loramendi 4, 20500, Mondragon, Spain [email protected], [email protected] decisions. However, in a product line, alternative design decisions and variant architectural styles can be applied to get products with the same functionality but different quality levels. During the software architecture evaluation, it must be ensured that all the products of the line will achieve the required quality attribute levels. Software architecture evaluation is defined as “the systematic examination of the extent to which an architecture fulfils requirements” [5]. To be able to analyse the potential of an architecture to reach the required quality levels helps to find the problems early in the life cycle; when they are easier and cheaper to correct than in the later stages such as in the implementation, testing or deployment stage. In the product-line framework, where a lack or mismatch in a quality attribute requirement is potentially replicated among all products, the ProductLine Architecture (PLA) evaluation is very important since problems can be detected before actual products are developed. Moreover, the life span of a PLA is much longer than the one of an ordinary software product and it serves as a basis for a set of related systems. In the case of PLAs the architecture assessment becomes crucial to ensure that the PLA is flexible enough to support different products and in order to allow evolution. The assessment of all the instances of the productline may not be worthwhile due to the high cost. However, it is possible to shorten product architecture evaluations because “The product architecture evaluation is a variation of the product-line architecture evaluation as the product architecture is a variation of the product-line architecture and the extent to which product evaluation is a separate, dedicated evaluation depends on the extent to which the product architecture differs in quality-attribute-affecting ways from the product line architecture” [4]. The workload and cost reduction of the evaluation of software product lines is here presented. The proposal is based on identifying the design features

Abstract Variability is a key aspect in software product lines. Functional variability has been largely studied as a way to obtain all the desired products for a line. Quality variability, less understood and more complex, has not received so much attention by researchers. However, different members of the line may require different levels of a quality attribute. The design phase is a good point to assure that quality attributes requirements are met within the product line so this means paying attention to software architecture evaluation during Domain Engineering. The quality evaluation in software product lines is much more complicated than in single-systems as products can require different quality levels and the product line can have variability on design that in turn affects quality. The evaluation of all the products of a line is very expensive. Thus, ways of reducing the evaluation efforts are necessary. Herein is presented a method for facilitating cost-effective quality evaluation of a product line taking into consideration variability on quality attributes.

1. Introduction The variability management in product lines must withstand functional and non-functional requirements. As stated by FODA: “A feature is a prominent or distinctive and user-visible aspect, quality, or characteristic of a software system or systems” [18], a feature can be functional or non-functional. In some domains the fulfilment of quality attribute requirements is even more important and more complex than addressing functional requirements. When developing embedded system product-lines time or safety issues may be critical. Quality attributes must have an important role during the design stage. In a software product line, as in single-systems quality attributes can guide design 1

This work was partially supported by The Basque Government (Leire Etxeberria enjoys a doctoral grant) and the Spanish Ministry of Science and Education under grant TIN2007-61770 (OPTIMA).

978-0-7695-3303-2/08 $25.00 © 2008 IEEE DOI 10.1109/SPLC.2008.37

243

An example based on the Arcade game maker pedagogical product line developed by SEI (Software Engineering Institute) [24] is used here as an example. The product line consists of three different games (Brickle, Pong and Bowling). Commonality in the product line includes a set of sprites (movable and stationary), graphical animation and a set of rules about the interaction of game pieces. Variability includes the type of sprites (puck, paddle, bowling ball, brick…), the number of sprites, the rules and the behaviour of sprites and the environment. Quality requirements are performance: the action of the game must be fast enough to seem continuous and usability: the game must be attractive for the player. The arcade product line has been previously evaluated using ATAM [3] and HoPLAA [23]. In these evaluations, indirect variation has not been considered. In our proposal indirect variation is used in order to reduce evaluation efforts. The rest of the paper is organized as follows. In Section 2 the quality specification model is presented, the method for quality evaluation is presented in section 3, in section 4 an example of the method is illustrated, in section 5 the validation of the approach is explained, related works are shown in Section 6 and finally in Section 7 the conclusions are presented.

that affect quality-attributes to reduce evaluation efforts. This information can help to apply effort reduction approaches. One approach is to create a generic evaluation model with variability that helps to evaluate the line reducing efforts, as several products can be evaluated together with that model. Another strategy is to select a subset of representative products or designs to evaluate and extrapolate the results to all the products. As many efforts have been dedicated to study product-line quality attributes [6] (that is evolution or development attributes that are inherent to product lines: extensibility, modifiability, etc.), the proposal focuses on the domain relevant quality attributes, those attributes that are execution or operational in the products such as safety in the safety-critical domain, performance in the real-time domain, reliability in the embedded systems, etc. Moreover, those are the quality attributes that are important at the product level and not only at the product-line level. In the evaluations of operational attributes in software product lines, a great range of products must be considered. Those products can differ by functionality, by quality attribute requirements (two products providing the same functionality but different quality) or probably by a combination of both. The idea of the variations in quality has been reported in different papers [14][21][20]. Different types of quality variability can be identified. Niemelä [21] defines three types of quality attribute variability: ƒ Variability among different quality attributes (optionality): Quality attributes only applicable to some products in the line. ƒ Different priority levels in quality attributes: The required level of a quality attribute can vary among products. ƒ Indirect variation (Impacts of functional variability on quality). Functional variability can indirectly cause variation in the quality requirements. Specifying and managing quality attributes, their variability (all types) and their relationship with functional aspects is the base for performing a costeffective evaluation. The information about the indirect variation helps to identify the variability that affects the quality attribute requirements. This way it is possible to construct a generic evaluation model or/and to focus on the most representative products to perform an evaluation. The method can be used at two abstraction levels: at design stage to evaluate product line architecture before implementing the line and at implementation level to evaluate the potential products of the line before generating all of them.

2. Extended feature model The Quality variability specification is the first step to assure that all the required levels of quality in all the products are achieved in the line. For that, all types of quality variability must be considered. Although quality attributes fit in the definition of feature [18], quality attributes have different and imprecise meanings depending on the domain [3] and a means for eliciting and refining quality attribute requirements is needed (quality attribute characterization). For this reason, the feature model has been extended with mechanisms from ATAM (Architecture Trade-off Analysis Method) evaluations [3] for representing quality attributes, their variability (optionality and levels) and the influence on quality of the functional, architectural and implementation features (indirect variation). The FeatuRSEB [13] approach has been integrated with an extension of the utility tree [3] that is used in the software architecture evaluations. The extended feature model [7] includes both functional and quality aspects and is used as a central element to capture the variability during the entire life cycle including design or implementation.

244

applied. This way, if instantiating the variability of the generic model and performing the evaluation is timeconsuming, only a subset of products is evaluated, reducing efforts. Which approach to apply, depends on the case, the quality attributes to evaluate, the evaluation moment, and so on. Sometimes, the construction of a generic evaluation model is not viable in a cost-effective way. In these cases, the selection of the products is the best option. If the product line is already implemented, the selection of the products to measure quality attributes via execution is also a good alternative.

Quality features of the model characterize quality attributes, concerns and scenarios (as stated in the utility tree) but also any other concept that can facilitate characterization, evaluation and derivation of quality in software product lines (such as quality levels used mainly with derivation purposes). See Fig 1 for an example. This approach is similar to Olumofin’s HoPLAA proposal [23] of introducing variability in the utility tree. We extend this approach to also address other quality variability types (such as indirect variation).

1. Construct Generic Evaluation Model

1. Select Products and instantiate

Product Line Architecture

2.

Generic model or

Generic model

2.

Fig 1: Characterization of Quality features Different types of variabilities are considered as follows: ƒ Optionality: The quality features can be optional, alternative or mandatory (as other features). ƒ Quality attribute levels: Levels can be defined in quality features. ƒ Indirect variation: the concept of impact (see Table 1) has been introduced in the feature model to explicitly define impacts among functional variants and quality aspects. Those impacts can be qualitative (defined by experts) or quantitative (resultant from evaluations).

… P1

Impact -Legend:

P2

P3

P4

… Pn

m products (m=n)

P1

P2

P3

P4

Pn

m products (mMobile (Pong, Low, noSound) PC--> Mobile (Pong, Low, Sound) PC --> Mobile (Pong, High, Sound) noSound --> Sound (Pong, PC) noSound --> Sound (Pong, Mobile) Low --> High (Pong, PC) Low --> High (Pong, Mobile) PC--> Mobile (Pong, High, noSound) Pong-->Brickles (High, Sound, Mobile) Pong-->Brickles (Low, Sound, PC) Pong-->Brickles (Low, noSound, Mobile) Pong-->Brickles (High, noSound, PC) …

2. Select products In order to select the products to be evaluated, the expected interaction degree is selected. For degree-2 and performance, applying the algorithm 12 products (see Table 5) from a total of 32 possible products are necessary to detect interactions of degree-2. Table 5: List of products # 1 2 3 4 5 6 7 8 9 10 11 12

Game type Pong Brickles Bowling SpaceWar Brickles Pong Bowling SpaceWar Pong Pong Pong Pong

Graphics resolution HighResolution HighResolution HighResolution HighResolution LowResolution LowResolution LowResolution LowResolution LowResolution HighResolution HighResolution HighResolution

Sound Sound Sound Sound Sound noSound noSound noSound noSound Sound noSound noSound Sound

Device Mobile Mobile Mobile Mobile PC PC PC PC Mobile PC Mobile PC

Time 5,2676 0,0229 4,79428 0,02109 30,42628 25,96748 4,04166 0,12206 10,93556 7,84876 1,71021 0,04461

Refresh time 4,77359 4,9999 5,2443 0,00221 0,22852 0,0004 0,2448 5,01799 25,15868 25,94458 -0,75262 0,10097

With those impacts, the refresh time of the limit products can be obtained: the product with the highest refresh time and the one with the lowest refresh time for instance. This way, the fulfilment of mandatory scenarios is checked. In the case of the new game: Spacewar in a mobile with Sound support and high resolution sprites, the refresh time is obtained adding the refresh time of the product with the lowest value (Pong, noSound, Low, PC) to the impacts of changing device, graphics resolution, sound support and game: 0,020069 + 4,77359 + 0,2448 + 0,22852 + 103,07544 = 108,342419; the time is higher that 100 milliseconds so it does not meet the scenario “Refresh every tenth of a second”. This value is obtained using the highest number of movable sprites and the highest number of possible collisions for that game (the worst case). As this game has considerably more sprites than the rest, it can not be assured that all the processing steps will finish, in the interval of 100 milliseconds.

3. Evaluation The evaluation model has been instantiated for those 12 products (applying the formula with the values of the Table 3) and the refresh time has been obtained for those products. Evaluation results have been compared for detecting interactions. For instance, if RefreshTime(Product1) − RefreshTime(Product9) ≠ RefreshTime(Product10) − RefreshTime(Product6) then there is an interaction

4. Add evaluation features In the case of usability, levels have been defined to help during derivation (see), the user can select a desired usability level and along with the quantitative impacts identify the required functional options that must be selected. Levels are: Low usability [0-33), Medium usability [33-66), High usability [66-100].

This way, several interactions have been detected. The number of interactions suggests that feature interaction degree is higher than two. If required, more products can be evaluated to determine exactly the interaction (which features interact and the degree). To detect degree-3 interactions 8 new products are required. The architectures of the 8 new products have been evaluated and the next degree-3 interactions have been detailed: Graphics resolution, game type and

249

5. Approach validation

+ Quantitative impacts Feature

Refresh time

PC -->Mobile (Pong, Low, noSound) PC--> Mobile (Pong, Low, Sound) noSound --> Sound (Pong, PC)

4,77359

The approach has been validated in several case studies (see table 7): ƒ A calculator product line where usability, efficiency and performance have been evaluated, as well as trade-off analysis made. The evaluation in this case study has been performed via product execution and no evaluation model has been constructed. This example is found in [8]. In this case study, the evaluation cost is reduced in a 98% because it is enough to evaluate 50 products of 2688. ƒ The Graph Product line (GPL) [1] is a known example, on which the approach has been also applied. The execution time and maintainability have been evaluated. This product line is implemented and execution has been used for testing products. In this case study, the evaluation cost is reduced 72,5% because it is enough to evaluate 44 products of 160. ƒ The arcade product line that has been presented in this paper. In this case, performance and usability has been evaluated and software architecture evaluation methods used for quality evaluation as code is not available. The use of a generic evaluation model reduces considerably the evaluation effort (up to 90%). ƒ Elevator product line is a case study on which we are working. In this case study, evaluation is performed at software architecture level and performance and reliability issues are evaluated.

4,9999 0,00221

…

Weights Levels

A weight is the contribution of a scenario, concern or other type of quality node to achieve its father node

Levels are defined to help during derivation. They represent a range of quality attribute values

Fig 8. Extract of the extended feature model with levels and weights Weights are also defined. Weights depend on how much the nodes help to obtain their father quality. See: Sound support has a weight of 30 and attractive graphics a weight of 70 to obtain usability. Example: A product with sound option (and low resolution) will have a usability of 100 on 30, so 30. During derivation automated reasoning can be applied, if High sound and Low resolution are selected, usability will be 30 (Low), so Low Usability will be selected automatically. If resolution is changed and High resolution is selected, usability will be (30 + 80 on 70 = 86), so High Usability will be selected automatically. The feature selection can also start by selecting a usability level: for instance if high usability level is required, Sound support and high resolution will be selected and, low resolution disabled. For performance, no groups or levels have been defined because it has only a mandatory concern and a scenario that always must be met. It is not necessary to define different levels of performance.

Table 7: Case studies and conclusions Case study Calculator

Quality attributes Usability, Efficiency, Performance

Effort reduction way Product selection

GPL

Execution time, Maintainability

Product selection

Arcade Elevator

Performance, Usability Performance, Reliability

Generic model Generic model and Product selection

During the application of the method in the case studies, it has been observed that to make the variability in quality attributes explicit helps to have quality in mind during all the life cycle (especially during Domain Engineering). Quality attribute requirements are make explicit and characterized and variability on design decisions and implementation is specified taking into account impact on quality.

Used evaluation techniques Quality Measurement (execution) Quality Measurement (execution) Software architecture evaluation Software architecture evaluation

Conclusions 98% of reduction 72,5% of reduction Up to 90% of reduction Evaluation and derivation time reduction

6. Related works Regarding quality specification, several FeatureBased Approaches that address quality variability modelling and reasoning can be found: Extended goalbased model [11], F-SIG Feature-softgoal

250

In this example, performance and usability have been considered but the method can be used with any quality attribute: reliability, security…The approach is focused on domain-relevant quality attributes but it can also be applied for evaluating development attributes specially when those attributes are important at product level. Moreover, the method can be also applied for evaluating quality via product testing (if the product line is already implemented) and not only with software architecture evaluation methods. The method uses the extended feature model to identify the variability that impacts on quality and this way facilitate the quality variability assessment (evaluate that all the required levels for a quality attribute in the line are achieved). To evaluate all the architecture instances of a product line is very expensive but with this approach it is possible to reduce costs. The method provides novel activities to reduce evaluation cost and effort and perform a costeffective evaluation of the product line. In this sense, there are three approaches in the proposal that are worth mentioning: ƒ Reuse of evaluation model: Create a generic evaluation model. ƒ Reduction of evaluation scope: Product selection algorithm ƒ Information for derivation In some cases, all the steps will be performed: create a generic evaluation model and instantiate it for the selected products. In other cases, such as the example illustrated above, a generic model can be created and all products can be evaluated without selecting some of them because the instantiation of the model is not time-consuming (only a formula needs to be applied). However, it is not always possible to create a generic evaluation model. For instance, for evaluating an already implemented product line, product execution can be used and in this case the only applicable step is the selection of the representative products. The proposed algorithm helps to select the most appropriate products and to detect the interactions and quantify the impacts of features on quality aspects. In order to select the most appropriate products to evaluate or create a generic evaluation model, qualityattribute-affecting features must be identified along with the feature interactions that affect those quality attributes. The extended feature model is not only used for facilitating the evaluation, it can be also used for facilitating a quality aware derivation. Feature model guides the selection of features to derive or configure a specific product. However, quality attributes or features are not usually among the features that guide this selection because quality attributes are not usually specified in a way that can be selected (they are at a

interdependency graph [17], Benavides et al. [2] approach, Zhang et al. [27] Bayesian Belief Network (BBN). There are also other methods that are not based on features: COVAMOF (ConIPF Variability Modelling Framework) [25] and Quality Requirements of a Software Family (QRF) method [22]. Nevertheless, existing approaches do not address all the requirements we consider necessary [9] and no method uses or proposes guidelines for using these variability models for evaluation. Regarding software architecture evaluation methods, there are some methods that are specific for product lines [6]: For evaluating product-line architectures there are different methods: FAAM [5] for evaluating information-system family’s architectures, the RAP method [16] for evaluating reliability and availability, the IEE method for integrability and extensibility evaluation [15] and DSAAM (Distributed SAAM) [12], a variant of SAAM for evaluating reference architectures. For evaluating both product line architecture and instantiated product architectures there is one method: HoPLAA [23], an adaptation of ATAM for product lines. This method introduces variability in the quality attribute utility tree used for evaluation. HoPLAA addresses the requirements for the evaluation of software product line architectures in an integrated, holistic approach. It is executed in two stages; the first stage focuses on the Core Architecture evaluation, while the second stage targets the evaluation of individual Product Architectures. Most of the methods focus on evaluating development quality attributes that are specific for product-lines (flexibility related qualities) at productline architecture level and there are few methods for evaluating run-time or domain relevant quality attributes (performance, reliability…) such as RAP. Our contribution is an approach that takes into account quality variability explicitly in order to evaluate domain-relevant quality attributes and reduce evaluation efforts (of any quality attribute) in software product lines. Our approach considers all the types of quality variability explicitly and it is generic; it can be used for any quality attribute and using existing architecture evaluation methods.

7. Conclusions and future work The approach has been applied using a specific evaluation method (PASA) but the method can be also used for any quality attribute and using other existing software architecture evaluation methods.

251

higher abstraction level). The proposed levels and impacts facilitate a quality aware reasoning and derivation. The required quality level can be selected and obtain which functional features can be selected and which ones not to. A reasoning based on propositional logic (as proposed by Batory [1]) is not longer enough and other reasoning techniques are required. Support tools for this method are being developed using CSP techniques as proposed by [2].

European Conference on Software Maintenance and Reengineering, CSMR, pages 354–363. IEEE Computer Society, 2005. [13] M. Griss, J. Favaro, and M. d'Alessandro. Integrating feature modeling with the rseb. In 5th International Conference on Software Reuse, ICSR, pages 76–85, 1998. [14] Günter Halmans and Klaus Pohl. Communicating the variability of a software-product family to customers. Software and System Modeling, 2(1):15–36, 2003. [15] K. Henttonen, M. Matinlassi, E. Niemela, and T. Kanstren. Integrability and extensibility evaluation from software architectural models - a case study. The Open Software Engineering Journal, 1:1–20, 2007. [16] Anne Immonen. Software Product Lines, Research Issues in Engineering and Management, chapter A Method for Predicting Reliability and Availability at the Architecture Level, pages 373–422. Springer, 2006. [17] S. Jarzabek, B. Yang, and S. Yoeun. Addressing quality attributes in domain analysis for product lines. IEE Proceedings - Software, 153(2):61–73, 2006. [18] K. Kang, S. Cohen, J. Hess, W. Novak, and S. Peterson. Feature-oriented domain analysis (foda) feasibility study. Technical Report CMU/SEI-90-TR-21, 1990. [19] Jia Liu, Don S. Batory, and Srinivas Nedunuri. Modeling interactions in feature oriented software designs. In Feature Interactions in Telecommunications and Software Systems VIII, ICFI, pages 178–197, 2005. [20] Varvana Myllärniemi, Tomi Männistö, and Mikko Raatikainen. Quality attribute variability within a software product family architecture. In 2nd International Conference on the Quality of Software Architectures (QoSA), 2006. [21] Eila Niemelä. Architecture centric software family engineering. Product Family Engineering seminars, Helsinki, Finland, 2005. [22] Eila Niemelä and Anne Immonen. Capturing quality requirements of product family architecture. Inf. Softw. Technol., 49(11-12):1107–1120, 2007. [23] Femi G. Olumofin and Vojislav B. Misic. Extending the atam architecture evaluation to product line architectures. In Proceedings of the 5th Working IEEE/IFIP Conference on Software Architecture, WICSA, pages 45–56. IEEE Computer Society, 2005. [24] SEI. Arcade game maker pedagogical product line. Arcade Game Maker Pedagogical Product Line. Web page: http://www.sei.cmu.edu/productlines/ppl/index.html, 2007. [25] Marco Sinnema, Sybren Deelstra, Jos Nijhuis, and Jan Bosch. Covamof: A framework for modeling variability in software product families. In 3rd International Conference on Software Product Lines, SPLC, Proceedings, pages 197– 213, 2004. [26] Lloyd G. Williams and Connie U. Smith. Pasa: a method for the performance assessment of software architectures. In Proceedings of the 3rd international workshop on Software and performance, WOSP, pages 179–189. ACM Press, 2002. [27] Hongyu Zhang, Stan Jarzabek, and Bo Yang. Quality prediction and assessment for product lines. In 15th International Conference on Advanced Information Systems Engineering, CAiSE, Proceedings, pages 681–695, 2003.

References [1] Don S. Batory. Feature models, grammars, and propositional formulas. In 9th International Conference on Software Product Lines, SPLC, Proceedings, volume 3714 of LNCS, pages 7–20. Springer, 2005. [2] David Benavides, Pablo Trinidad, and Antonio RuizCortés. Automated reasoning on feature models. In Advanced Information Systems Engineering, 17th International Conference, CAiSE, Proceedings, pages 491–503, 2005. [3] Paul Clements, Rick Kazman, and Mark Klein. Evaluating Software Architectures: Methods and Case Studies. Addison-Wesley Professional, January 2002. [4] Paul Clements and Linda Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley Professional, August 2001. [5] Thomas J. Dolan. Architecture Assessment of Information-System Families: a practical perspective. PhD thesis, Tech. Univ. Eindhoven, Netherlands, 2001. [6] Leire Etxeberria and Goiuria Sagardui. Product-line architecture: New issues for evaluation. In 9th International Conference on Software Product Lines, SPLC, Proceedings, volume 3714 of LNCS, pages 174–185. Springer, 2005. [7] Leire Etxeberria and Goiuria Sagardui. Evaluation of quality attribute variability in software product families. In 15th IEEE International Conference on Engineering of Computer-Based Systems, 2008. [8] Leire Etxeberria and Goiuria Sagardui. Quality assessment in software product lines. In 10th International Conference on Software Reuse, ICSR, volume 5030, pages 178–181. Springer, 2008. [9] Leire Etxeberria, Goiuria Sagardui, and Lorea Belategi. Modelling variation in quality attributes. In 1st International Workshop on Variability of Software-Intensive Systems, VaMos, volume Lero Technical report 2007-1. Lero, 2007. [10] E. Folmer, J. Gurp, and J. Bosch. Scenario-based assessment of software architecture usability. In Proceedings of Workshop on Bridging the Gaps Between Software Engineering and Human-Computer Interaction, ICSE, Portland, 2003. [11] Bruno González-Baixauli, Miguel A. Laguna, and Yania Crespo. Product line requirements based on goals, features and use cases. In International Workshop on Requirements Reuse in System Family Engineering (IWREQFAM), pages 4–7, jun 2004. [12] Bas Graaf, Hylke van Dijk, and Arie van Deursen. Evaluating an embedded software reference architecture: Industrial experience report. In Proceedings of the 9th

252