Improving software product line configuration: A quality attribute-driven ...

Information and Software Technology 55 (2013) 541–562

Contents lists available at SciVerse ScienceDirect

Information and Software Technology journal homepage: www.elsevier.com/locate/infsof

Improving software product line configuration: A quality attribute-driven approach Victor Guana a, Dario Correal b,⇑ a b

Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada T6G 2E8 Systems and Computer Engineering Department, Universidad de Los Andes, Cra 1E N 19A-40 Bogota, DC, Colombia

a r t i c l e

i n f o

Article history: Available online 2 October 2012 Keywords: Software architecture Model driven – software product lines Variability management Quality evaluation Sensitivity points Domain specific modeling

a b s t r a c t Context: During the definition of software product lines (SPLs) it is necessary to choose the components that appropriately fulfil a product’s intended functionalities, including its quality requirements (i.e., security, performance, scalability). The selection of the appropriate set of assets from many possible combinations is usually done manually, turning this process into a complex, time-consuming, and error-prone task. Objective: Our main objective is to determine whether, with the use of modeling tools, we can simplify and automate the definition process of a SPL, improving the selection process of reusable assets. Method: We developed a model-driven strategy based on the identification of critical points (sensitivity points) inside the SPL architecture. This strategy automatically selects the components that appropriately match the product’s functional and quality requirements. We validated our approach experimenting with different real configuration and derivation scenarios in a mobile healthcare SPL where we have worked during the last three years. Results: Through our SPL experiment, we established that our approach improved in nearly 98% the selection of reusable assets when compared with the unassisted analysis selection. However, using our approach there is an increment in the time required for the configuration corresponding to the learning curve of the proposed tools. Conclusion: We can conclude that our domain-specific modeling approach significantly improves the software architect’s decision making when selecting the most suitable combinations of reusable components in the context of a SPL. Ó 2012 Elsevier B.V. All rights reserved.

1. Introduction Reusability strategies are strongly related to the study and development of software product lines (SPLs) [1–5]. SPLs provide a technique in which an application domain manages reusable software elements to construct a software product family. From an application engineering perspective, the basic reusability strategy in the software product lines is based on the reutilization of two principal assets: the software architecture of the line (SPLA) that defines the family of products that belong to the SPL, and the software artifacts that provide the architecture to build a new product [6]. In this work, we align our proposal with the Framework for Software Product Line Practice, version 5.0 (FSPLP) of the Software Engineering Institute (SEI) [7]. In the FSPLP, three essential activities are required for building SPLs: core asset development, product development, and management. The core asset development activity establishes a production capability for the products. This ⇑ Corresponding author. Tel.: +57 1 3394949. E-mail addresses: [email protected] (V. Guana), [email protected] (D. Correal). 0950-5849/$ - see front matter Ó 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.infsof.2012.09.007

activity has as outputs three main artifacts: the product line scope, the core asset base, and the production plan. The product line scope contains the description of the products to be produced by the product line. The core asset base includes an architecture shared by the products of the SPL, and a set of software components developed for systematic reuse in the SPL. The production plan prescribes how the products are produced from the core assets base. The production plan needs to be instantiated into a product production plan, describing how an specific product will be derived using the core assets base [7]. During the product development activity, product builders use the core assets, according with the product production plan and a description of the particular product to be build, to create a new product that meet specific functional and nonfunctional requirements. The Management activity is related to the technical and organizational tasks required to operate the product line, like risks management, definition of a funding model, and production tracking process [7]. The FSPLP defines three product line practice areas required to perform the activities mentioned above. These practice areas are

542

V. Guana, D. Correal / Information and Software Technology 55 (2013) 541–562

classified into three categories: Software engineering, technical management and organizational management. Software engineering practice areas are related to the definition and evaluation of the architecture of the SPL, the development of the components, and the management of the requirements. The technical management practice areas deal with the configuration management, measurement, tracking, and scoping of the SPL. Specifically in this category there is a practice area called Technical Planning (TP), in which product production plans are defined, describing the production process for a particular product. According with the FSPLP, a good product production plan should answer the question of what core assets need to be used or adapted to support a particular product requirements. Organizational practice areas, include tasks like business case, funding, market analysis, operation and training tasks [7]. 1.1. Problem statement During the elaboration of a product production plan, product builders need to evaluate many possibilities to select the components that appropriately fulfil a product’s intended functionalities, including its quality requirements (i.e., security, performance, scalability) [8]. Incorrectly selecting reusable assets may allow a product that cannot meet its expected quality properties [9,10]. Even though a SPL has a predefined production plan for the assemblage of candidate products, quality attributes such as response time or memory consumption may vary according to each product specific context. This behavior makes the specific component selection a vital task from a quality fulfilment perspective. Moreover, as the product line matures, product line components grow in number, making the elaboration of a product production plan a cumbersome task. Product builders lack tools to support the selection of appropriate assets fulfiling functional and non-functional requirements during the definition of the product production plan for a particular product [11]. 1.2. Goals and contributions We formulated three research questions to address the problem in this work. RQ1: Can the implementation of a domain specific modeling strategy improve the representation of the information that constitutes the assets of a SPL? RQ2: Can implementing a model analyzer improve a product builder’s decision making when selecting the most suitable combinations of reusable components on a SPL product production plan? RQ3: Does the mentioned analyzer improve component selection with a set of quality requirements as input? While question RQ1 is framed in the domain engineering process of the SPL, questions RQ2 and RQ3 are focused on the application engineering process of the product line. We present a Model-Driven Engineering (MDE) approach to give the product builder a set of tools to automate, simplify, and reduce errors when defining an SPL product production plan. By resolving this problem, the product builder can more effectively reutilize assets that can structure the SPL’s architecture to get a high-quality product. Our proposal contributes the following elements: (i) a set of metamodels that define well-formed languages for the expression of the SPL constitutive domains, (ii) a Domain-Specific Language (DSL) to express critical points on the SPLA, (iii) a model generation engine that uses the SPL domain models and the critical points on the SPLA to create a unified model of the SPL information, (iv) an analysis engine that takes an SPL unified model as input and pro-

duces reports about expected behavioral responses of the SPL, and (v) a conceptual framework that formalize the modeling and filtering evaluation techniques on the definition SPLs production plans. With our tool and framework, we pursue scoring the possible SPL assets, prioritize them, and create reports that aid the product builder when defining a product production plan for the derivation of a particular product. 1.3. Paper structure This article is structured as follows. Section 2 presents a case study to illustrate the posed problem and validate our solution. A formal specification of our proposal conceptual framework is portrayed in Section 3, Section 4 presents an overview of our strategy. Section 5 exposes the model-driven approach used here. Section 6 explains how we compose different heterogeneous models into one unified model. Section 7 elaborates the analysis algorithm that selects the appropriate configuration of an SPL. Section 8 validates our case study. Section 9 discusses our proposal threats to validity. Section 10 presents the related work. Finally, Section 11 presents our conclusions and future work. 2. Motivating case study In 2008, the Fundacion SantaFe de Bogota Hospital (FSFB), in conjunction with Los Andes University, started a project to offer a tele-consultation service to the community based on the use of mobile technologies. From the beginning, the objective was to define a Software Product Line (FSFB-SPL), capable of deriving different tele-consultation products, sharing a common architecture and reusing a set of components developed to this end. Using this SPL different products has been derived during the last years, including a general purpose tele-consultation application called Dr.Chat Mobile. A diabetes and overweight tele-consultation application, a headache tele-consultation application, and lastly a tele-consultation application for sexual education and orientation. The architecture of the FSFB-SPL is based on a client–server style, where the mobile application serves as a client to send health consultations to a medical center (server side). In a typical scenario, a patient who presents disease symptoms uses a cellphone with the mobile application to initiate a consultation with a medical professional. The patient can describe symptoms using text, pictures, voice, or videos. The consultation then travels to the Hospital server, which stores user information. At the medical center, physicians can submit particular responses to the consultations. The patient can then asynchronously check if the medical center has answered his or her inquiries. The SPL has a component-oriented architecture as shown in Fig. 1. It has nine main components: the Main Controller, which handles GUI (Graphic User Interface) events from the user; the Media Controller, which wraps the application’s media management capabilities from the Audio, Video and Image Managers; the Communication Manager, which handles the communication protocols and mechanisms to send and retrieve information from the medical center consultation services; the Persistence Manager, which stores consultation records and attached media files with a patient’s medical history; and the Localization Manager component, which tracks a patient’s position to introduce geolocalization references to the consultations. To express our problem’s variability, we use the feature modeling presented in [12]. Fig. 2 defines our case study variability model through a feature model approximation. The SPL can produce a remote medical consultation application with the following features: the mandatory communication, persistence, and information input features, and an optional localization feature that


543

Fig. 1. Mobile tele-consultation SPL component and connector model – functional view.

Fig. 2. Dr.Chat variability expression – feature model.

allows the application to introduce geo-references into a patient consultation. On a second level of variability, a user can configure the localization feature to be automatic or manual. According to the model, the application communication feature can be configured as secure and its communication channel alternatively configured with local (WiFi, Ad hoc) or mobile (cellular) networks. The information input also varies in the media information inputs through which a user can describe the relevant pathology, whether plain text, video, image/photography, or audio. To meet this product line variability expression, we have been maintaining a basic set of reusable components part of the core assets base of the SPL architecture. These components have been designed to realize variation points inside the architecture according to rules determined by the product line variability models. Specifically, 36 concrete components can realize the communication variation point in the architecture. There are two possible components for local communication, Wi-Fi local and Ad-hoc, and two for mobile communication, EDGE mobile and 3G. Additionally, each communication component has a secure variation. Each implementation includes one of four available cipher algorithms: AES, Blowfish, DES, and Triple-DES (including a cipher key size specific to the selected algorithm). There are four image manager components, a single plain text information input component, three audio manager components, and three video manager components. The different audio, video, and image components vary in media information compression and quality levels. The product line core asset base exposes five possible concrete components that can implement localization variation points in the architecture localization abstract component. These are the automatic and manual localization components. A WiFi Based triangulation component and a Global Positioning System (GPS) constitute the first set. The second set is structured by different components that allow the user to manually select a location from a geo-reference point database. The persistence component implementations vary in their included cipher algorithms and key sizes. The persistence components include a non-secure component, called Plain Registry Persistence, and nine secure components with different cipher algorithms and key sizes (i.e., AES, Blowfish, DES, Triple-DES). In summary, the product line has 11 external variability points, an

architecture with nine abstract components (internal variability points), and a component catalog of 64 components. In total, it can be configured in 963,360 ways answering different product characteristics. 3. Conceptual framework Our conceptual framework involves a subset of the SPL core assets to simplify the elaboration of product production plans. This subset of assets, called configuration assets, involves (i) the software architecture that the products share, (ii) the variability models that specifies the scope of the line, and (iii) the concrete artifacts that realize the architecture abstract elements. From our perspective, the software architecture of the SPL is defined with architectural models and structured using views (i.e. functional, module, and allocation) [13]. The product line variability models are expressed through the feature modeling technique in order to specify, in a hierarchical manner, the possible variable features that a product can hold within a SPL configuration. The concrete artifacts are defined as the executable set of components that can implement the role of an abstract component inside the product line architecture, and realize a variation point of the SPL definition (see Fig. 3). The role of the SPL product builder is to choose the variability model features that have to go inside a product to derive. Below, we present the conceptual framework in which we based our strategy, Section 4 explores the tools we have developed to concretize our framework. Lets define four sets that formalize the core assets of a SPL and the expected quality attributes of a product to derive. Let A : fa:1; a:2; . . . ; a:ng be the set that represents the SPL architecture, thus a.i an abstract component of the SPL architecture. Let C : fc:1; c:2; . . . ; c:ng be the set that represents the components in the SPL core assets base, thus c.i a concrete component of the SPL. Let V : fv p:1; v p:2; . . . ; v p:ng be the set that represents the SPL variability, thus vp.i a variation point in the variability expression of the SPL (see Fig. 4).

544


Fig. 3. Software product lines configuration assets relations.

Finally, Let Q : fqs:1; qs:2; . . . ; qs:ng be the set that represents the expected quality attributes of a product, thus qs.i a quality scenario that a product to derive must fulfil. The realization relation c : C V ! B is defined as the property of a concrete component c.i to realize a variation point vp.i. The implementation relation a : C A ! B is defined as the property of a concrete component c.i to implement an abstract component a.i in the architecture of the SPL. Additionally, let k : C A ! Q be the relation in which the realization of an architecture abstract component a:i 2 A with a concrete component c:i 2 C; aðc:i; a:iÞ ) Q . It means that the aðc:i; a:iÞ fulfils a product with the expected quality scenario q:i 2 Q in the architecture of a system (see Fig. 5). Let K # C be a subset of components in the core asset base, and W # A a subset of the SPL architecture (abstract components). A product production plan S : fV; K; Wg is the set of related elements in which:

Fig. 5. Formal realization (gamma) and implementation (alpha) relations.

We have defined a quality product production plan set SQ : fV; K; W; Q g as a set where, given a subset of the SPL component catalog (concrete components) QK # C,

ð8v p; co; aojv p 2 V; co 2 QK; ao 2 W : ð9c; ajc 2 QK; a 2 W : cðc; v pÞ ^ aðc; aÞÞ ^ ðkðco; aoÞÞÞ is true:

ð8v pjv p 2 V : ð9c; ajc 2 K; a 2 W : cðc; v pÞ ^ aðc; aÞÞÞ This is, for all the variation points in V, there is at least one tupleðc; aÞ from which c can realize the variation point cðc; v pÞ, and c implements at least one of the abstract components in the architecture aðc; aÞ. However, the main problem that arise with a product standard production plan is that it does not include the quality attribute set as a part of the configuration of a product to derive. Furthermore, our research experimentation (see Section 8) allows us to identify that for the elements of S,

ð8v p; co; aojv p 2 V; co 2 K; ao 2 W : ð9c; ajc 2 K; a 2 W : cðc; v pÞ ^ aðc; aÞÞ ^ ðkðco; aoÞÞÞ is false It means that not all of the concrete components can fulfil the quality scenarios for a product to derive, even when they can realize a variation point and implement an abstract component in the architecture.

Fig. 4. Architecture, components, and variability set definition.

Unlike S, in SQ the combination of the components that realize the variation points, and implement the architecture, meet the quality scenarios in Q. What we want to support is the filtering process of the elements from K, that conform QK. That is, the subset of concrete components that can realize a variation point, implement an abstract component in the product architecture, and additionally, fulfil the set of expected quality scenarios for the product to derive (see Fig. 6). As a pivotal point of our proposal we have used the sensitivity points in order to highlight critical points in the relations of the SPL core assets that determine the accomplishment of an specific quality scenario. An SP is defined as the set SP : fcðc:i; v p:kÞ; aðc:i; a:jÞ; q:i 2 Q g. In this set, a realization and implementation relation that are related with the satisfaction of kðc:i; a:jÞ for an specific q.i in Q.

Fig. 6. Implementation – quality fulfilment relation.


545

Fig. 7. Sensitivity point conceptual definition.

As an example imagine a quality scenario that demands a maximum memory consumption of 5 MB for the persistence component of a Dr.Chat Mobile product, thus a hypothetical SP will be defined as the realization of the variation point ‘‘PersistenceSecure’’ by either the ‘‘PMSecureAES128’’, or ‘‘PMTDES192’’ concrete components in order to implement the architectural abstract component ‘‘Persistence Manager’’, and additionally, imply the resolution of the quality scenario ‘‘Total Memory Consumption’’ (see Fig. 7). Our proposal involve the expression of the A, C, V, and Q sets as models using Model-Driven Engineering approach. Additionally, we want to provide an expression mechanism to built the elements and relations that constitute the SP. We produce a conciliation model that is then automatically analyzed pursuing the resolution of an optimal set of components to be used in the product production plan. 4. Strategy overview The Sensitivity Analysis Framework (SPAF) is a Model-Driven framework developed to support a product builder during the product derivation activities in an SPL. The strategy behind the SPAF framework supports three main activities: (i) modeling the SPL core assets, (ii) modeling architectural sensitivity points and unifying these models into a conciliated model, and (iii) analyzing the conciliated model to select the most appropriate set of components for the SPL architecture implementation. Our proposal process is portrayed in Fig. 8. 4.1. SPL core assets modeling overview The first activity, modeling the SPL’s core assets, uses a domainspecific modeling approach [14–18], this process is located in the domain engineering process of the SPL. It is specifically placed within the domain design and domain implementation subprocesses. Using our proposal, we express the SPL software architecture (SPLA) and its variation points through two existing metamodels for variability and software architecture description. Additionally, we express the reusable components with a core asset metamodel. We have also designed a domain-specific language to express a product’s derived quality requirements such as security, scalability, or performance. The output of this first stage is a

set of well-formed models describing the SPLA, the external variability of the SPL as a set of application features, the SPL set of concrete core assets (i.e. components), and the expected quality attributes of a specific product to derive. The last model has to be produced every time a new product with different expected quality attributes is going to be configured and later derived. 4.2. Sensitivity point modeling overview A domain-specific language is used to define critical decisions that the architect makes during the software product architecture design. Initially defined by Bass et al. [13], these decisions are sensitivity points. They represent critical aspects in a software architecture that are known because of their impact in the fulfilment of a software system quality requirements. From our SPL perspective, the product line sensitivity points are based on the realization and implementation decisions that a product configuration manager has to take in order to specify and derive a product. The following statement is a example of a sensitivity point in our SPL case study presented in natural language. Given the variability rules of the line, and the total memory consumption quality scenario for the ‘‘store consultation’’ operation, the realization of the variation point ‘‘persistenceVP’’ has to meet, through its realization, the response measure of ‘‘12 MB of memory consumption’’ with a concrete component implementation in order to fulfil the abstract ‘‘PersistenceManager’’ component role in the architecture. In this example, and following our framework, the realization of the persistence variation point is critical in the architecture due to the SPLA characteristics around the persistence abstract component to implement, and the related implications on achieving a quality requirement such as a memory consumption restriction; this decision responds to a sensitivity point in the SPLA. The outputs of this activity are a textual model of the SPLA sensitivity points, a conciliated model that gathers the set of models containing the core assets of the SPL (Activity 1 output), a model showing a product to derive quality requirements, and a model representing critical decisions (sensitivity points) that the architect makes when designing the SPLA [19]. The conciliation model is produced using a model weaver. This model makes explicit the relationships between the SPLA, the components model, the

546


Fig. 8. SPAF Execution process.

quality attribute requirements model and the sensitivity point model. This step simplifies instantiating the SPLA from its functional and quality requirements. Our complexity reduction strategy is based on the application of the conceptual framework presented in Section 3. Using this framework, we implemented the conciliation algorithm, allowing us to automatically obtain all possible component combinations required to fulfil a product’s functional and quality requirements. Taking the architectural sensitivity points as evaluation pivots, we reduce the problem searching complexity approaching the model elements in an filtered ordered set. 4.3. Conciliation modeling overview The third step in our strategy analyzes the conciliation model to assist the software architect when deriving a product. SPAF analyzes all possible concrete component combinations stored in the conciliation model. This step outputs a report evaluating all suitable possibilities from a functional and quality perspectives. This report is used by the product builders to define the concrete components to be used in the product production plan. In the following sections we further explain each of these activities in detail. 5. SPL core assets modeling This stage primarily aims to create an information repository following a Domain-Specific Modeling (DMS) approach. Even though software product lines are composed by management processes, and product release plans, among many others, the configuration complexity problem can be reduced to a sub set of product line assets inside the product engineering phase of a SPL derivation process. To evaluate possible SPL production plans for product derivation, we must first know the architectural structures that define the family of products (SPLA), how the SPL varies in the domain features inside a product configuration (SPL variability models), and the available concrete components that can recognize a variation point and implement an abstract architectural element inside the SPL architecture. The information model must expose the different behavioral characteristics of the SPL elements. We consider relevant the architecture of the product line, its core asset base,

and its variability models. To express each information domain, we developed and/or used the following domain languages defined through metamodels. 5.1. SPLA modeling Modeling the SPL architecture (SPLA) exposes structural relations between the concrete components of its products and their external related entities (non-software units, external systems, among others). An architectural model presents the restrictions and characteristics of the product line abstract architectural elements, their relations, and their particular properties. We believe that SPL architectural models provide the information required to validate the quality scenarios that a specific derived product must have. We use a view/perspective architectural analysis to model architectures in a concern-specific manner. We consider the SPLA as a set of models that belong to a family of views under three principal domains [13,20] (i.e., the execution, development, and allocation domains). We used Archivol [21,22], a domain-specific metamodel specified for architectural system descriptions. The Archivol metamodel provides three main packages: Architecture, Candidate, and Quality. The first package contains the language core meta-concepts, the second contains the main architectural element and relation concepts, and the third contains the quality-concerned constructs. An architect can express any software architecture following a set of style architectures, including client–server, n-tier or service-oriented architectures. Fig. A.17 (A) presents an example of a subset model containing an architectural view of Dr.Chat Mobile using Archivol. A more complete description of the metamodel can be found in [22]. 5.2. Core asset components modeling Modeling the components in the core asset base expresses the existing SPL contracts that a set of concrete components can implement. The model also states the components’ implementations, restrictions, and behavioral properties. We designed the Concrete Asset Metamodel (CAMM) to specify reusable sets of concrete assets (see Fig. 9). The main concept of CAMM is the ReusableSet. This concept groups the functional contracts that constitute the SPL’s set of


547

Fig. 9. Concrete Asset Metamodel (CAMM).

reusable packed functionalities. The concept Contract models a set of capabilities coherently grouped according to their functional domain. The AbstractCapability concept represents (from a functional perspective) an operational contract, including its related Values (inputs/outputs). Each contract has one or multiple Implementations. This last concept demonstrates a concrete artifact that meets a specific contract. It implements the set of capabilities according to its parent contract relations. A Capability specifies a concrete operation implementation, including its behavioral properties with the BehavioralProperty concept. The Implementation concept also includes a set of Restrictions as behavioral characteristics that must be provided for the implementation to work properly. Both the Restriction and BehavioralProperty concepts are related to a Measure element. It allows modeling quantifiers over their related parents. Fig. A.18 portraits an example of a concrete component subset model using CAMM. 5.3. Variability modeling There are numerous approaches to specify varying features and requirements in the SPL context [23]. Almost all of them use the orthogonal variability modeling (OVM) proposed by Pohl et al. in [3] to describe the SPL variability without extending its architectural models with new notations. Particularly, our proposal follows Pohl’s proposal by modeling the external variability of the SPL which has to be realized by internal variation points or concrete SPL assets. We used the Feature Metamodel (FMM) presented in [12] to model the SPL’s external variability or scope (see Fig. 2). This metamodel approximates FODA [24] to express variation points and relations over the SPL’s features. However, we have implemented a simplified version of the referenced metamodel in order to exclude different types of containment relations and attribute definitions that are out of our problem scope. Fig. A.19 exemplifies the communication feature branch of the Dr.Chat Mobile variability model using the FMM. 5.4. Product quality scenario modeling To model product expected quality scenarios, we used the Archivol’s Quality Package. This package allows us to model quality

attributes and quality scenarios [25] to specify expected behavioral characteristics of a derived product. Fig. 10 shows a performance quality attribute model conforming to the Archivol metamodel. It includes the memory consumption and response time quality scenarios for a particular Dr.Chat Mobile product. 6. Sensitivity point modeling and conciliation model weaving Once the architect has created the SPL models and the expected product quality scenarios, the next step is defining critical points inside the line models that are crucial for evaluating stimulus responses of the system under construction. From an SPL perspective, product line sensitivity points are based on the realization and implementation decisions made by a product configuration manager to specify and derive a product. This stage aims to build a unified model that gathers information inside different models to evaluate and select the appropriate components for a product production plan. The sensitivity point modeling stage has two main steps: (i) the sensitivity point definition, and (ii) the conciliation model generation. 6.1. Sensitivity point definition To identify SPL’s realization and implementation relations and associate these relations with the critical points for a product’s quality fulfilment, we designed the sensitivity point definition language (SPDL). In this language, the realizations, implementations, and their connections to a derived product’s quality scenarios are called binding points. A complete description of the design and implementation of the SPDL, including an editor developed for it, can be found in [26]. The binding points are (i) the variation points (from the variability model of the line) realized by concrete components (from the SPL component model), (ii) for these realization relations, which are its related quality scenarios (a scenarios impacted artifact related to a realization relation), and (iii) for each specific scenario, which includes the impacted views of the SPL architecture that hold scenario evaluation information. Two sections characterize the SPDL: the first is the variation realization section. Here, the architect can express the relations of

548


Fig. 10. Dr.Chat Mobile performance quality scenarios model (partial view) conforming to Archivol.

the concrete components that can realize a variation point inside the variability feature tree. The relations can be one-to-one, or one-to-many in any level of the tree. The second one is called the analyzed quality attribute section. Here, the architect can expose a set of quality scenario execution locations during evaluation. The entire textual model has been inspired by the sensitivity point terminology used in [6]. The textual representation is called sensitivity binding model or SBM. Listing A.21 presents an SPDL snippet to define a SBM in our Dr.Chat Mobile case study. This concrete example defines a sensitivity binding model and highlights the variation point realization relations using the concrete components ‘‘PMSecureAES12’’ and ‘‘PMSecureTDES192’’.

In this case, the architect intends to analyze the ‘‘MemoryConsumptionScenario1’’, and ‘‘ResponseTimeScenario1’’ regarding the possible variation point realization relations in the SPL. A complete SPL sensitivity binding model definition would involve all the SPL variation point realizations and set of product specific expected quality attributes. For simplicity in our example, we have only used two variation realization relations and two quality attributes to examine. Using the SPDL, we defined the variation realization section with the reserved word VariationRealization. Line 5, for example, defines the ‘‘persistenceSecureVP’’ variation point realization (from the product line’s variability model) by the concrete component ‘‘PersistenceManagerSimple’’ (from the SPL’s component catalog model).

Listing A.21. Example of sensitivity binding model using the SPDL.


The analyzed quality attribute section is defined in consecutive blocks using the reserved word AnalizedQualityAttribute. In our example, this section is defined on lines 9 to 25. Each analyzed quality attribute contain one or multiple sensitivity points in the SPLA where the quality scenario finds its execution. For example, a sensitivity point for the ‘‘MemoryConsumptionSenario1’’ is defined in lines 11 to 15. In the definition, two main items are specified. First, the impacted architectural artifact, or abstract component where the execution of the scenario takes place, and second, the impacted views, or architectural models, where the composition algorithm will find the related behavioral information for the quality evaluation of the concrete components that realize a selected feature in relation with the SPLA characteristics and restrictions. By indicating the impacted architectural view, the composition algorithm, and later, the evaluation engine, will determine only the valuable information for the scoring of the quality scenario in terms of the configuration context. While a dynamic quality scenario will take the architectural element properties of the ExecutionComponent model, a sensitivity point defined in terms of a deployment quality scenario would be evaluated using only the ComponentDeployment architectural model properties. The resulting SBM conforms to the SPDL’s metamodel given by the language grammar (see Fig. 11). 6.1.1. Conciliation model The next step in our strategy, using the SBM created in the previous step, a product quality scenario model, and the SPL asset models, generates a unified model to evaluate multiple SPL configuration decisions. We called this generated model the Conciliation Model (CM). This step aims to gather information scattered throughout the product line’s definition models in a single unified model. Given our domain-specific approach, we modeled each information domain using a language specified for it (i.e., SPL’s variability, architecture, concrete assets, and expected product quality attributes). To provide a unified information repository for line configuration variability evaluation, we supplied a composition algorithm that combined the different SPL model elements to create a Conciliation model following the relations exposed in the SBM as the composition guidance.

549

To generate the conciliation model in a well formed language, we defined the Conciliation Metamodel (CMM) presented in Fig. 12. This metamodel allows us to gather information from the SPL’s models in a single unified source of information. The conciliation metamodel contains concepts from the expected product quality scenario, the sensitivity point definition, and all SPL definition domains. In the remainder of this section, we first present the CMM structure and then a method for generating a conciliation model based on the SPL core asset models and SBM. The CMM primarily relies on the SensitivityModel concept. This element represents the set of all sensitivity points declared by the SPL architect inside the sensitivity binding model. Each sensitivity point has related quality scenario parts such as Stimulus, Response, Load, ImpactedViews, related VariationPoints, and impacted ArchitecturalElements. The ArchitecturalElement concept has both the Contracts that it must fulfil and all possible concrete Implementations that the SPL has offer in order to implement its role. The ArchitecturalElement also has its related AttachedProperties found in all related architectural models declared as impacted views. The Contract contains its related execution Restrictions and the BehavioralProperties of the Capabilities modeled as behavioral characteristics. 6.1.2. Conciliation model composition Weaving a conciliation model gathers information from the SPL domains. The SBM provides primary guidance for model composition. In the SBM, the architect expressed the realization relations and critical points to achieve quality scenarios in the product architecture. The composition scheme must first find all possible implementation relations, build the declared sensitivity points around them, and then activate those deemed viable by the SPL external configuration. A step-by-step SPM composition algorithm is shown in Fig. 13. The first step introduces five models to the model composition context: (i) the SPL architecture model (conforming to the Archivol metamodel), (ii) its variability model (conforming to FMM), (iii) its component model (conforming to CAMM), (iv) a SBM (conforming to the SPDL metamodel), and (v) a seed model (conforming to CMM).

Fig. 11. Sensitivity point definition language DSL’s metamodel.

550


Fig. 12. Conciliation Metamodel (CMM).

Th second step is finding implementation relations. The composition algorithm looks for implementation relations. Given all ArchitecturalElement [type: component] concepts in the SPL architecture model, the engine looks for all possible Implementations concepts (from the component model) that can implement each ArchitecturalElement. In the component model, if an Implementation concept belongs to a set of Contract concepts that contains the same names as the related ConnectionPoint [source:false] ArchitecturalElement concepts under evaluation, that tuple is considered a prospective implementation relation. The engine also looks for each AbstractCapabilities concept in the Contract set and verifies whether they match by name, quantity, and related Value concepts (attributes, relations, and concepts respectively). The engine verifies whether these concepts match the ones related in the ConnectionPoint [source:false] Capabilities. Using this selection mechanism,

the engine is capable to find all possible concrete components that can implement an abstract component in a SPL by name and structure. The third step is finding the realization relations. The architect explicitly expresses the realization relations in the SBM, because the SPL’s variability domain is semantically far from the architecture language abstract syntax (i.e., they cannot be inferred without a third information source). In Listing A.21, the variation realization section exposes which variation points of the SPL variability model (FeatureNodes) are realized by concrete components from the SPL catalog (Implementations). This uses the elements’ ids as matching tuples for the realization relations. The fourth step is loading the SensitivityPoint definitions. This step is the biggest in the composition algorithm and is divided into four sub-steps. For the final model, each step builds SensitivityPoint

Fig. 13. Conciliation model composition algorithm.


concepts from its simplest constructs, to the complex ones in a recursive manner. The analyzed quality attribute section of the SBM provides critical composition guidance for this particular step. During step 4a, the algorithm creates a SensitivityPoint for each SensitivityPoint concept declared inside the AnalyzedQualityAttributes of the SBM. Using the QualityScenario pointer declared in the SBM, the engine relates the SensitivityPoints to the stimulus and expected responses from the referenced AttributeRequirement construct. During step 4b for each referenced AttributeRequirement concept in a SensitivityPoint declared in the SBM, the algorithm extracts its Target concept and, from this, its related ArchitecturalElement (located in the SPL architecture model). It relates the last concept to the corresponding SensitivityPoint in the conciliated model. In step 4c the realization relations are loaded into the composition model. The engine examines the component catalog model for the existing ArchitecturalElements’ Contracts in the destination model and, using the relation tuples (calculated in step 1), adds all possible concrete components (Implementations) to the model. During step 4d and according to the ImpactedViews declared inside the AnalyzedQualityAttributes in the SBM, the engine examines the architecture model for all ArchitecturalElementProperties attached to the ArchitecturalElement referenced in each SensitivityPoint. The engine then adds the properties to the ArchitecturalElements in the final composition model. The fifth step activates the realization relations. According to the product line variability model, not all variation points must be evaluated for product configuration, because many of them were not selected for the external configuration of the product under construction. In the final model, the engine therefore sets the attribute [active:¼true] in the SensitivityPoints that, as impacted elements, have concrete components that realize a [selected:¼true] FeatureNode in the SPL’s variability model. Fig. A.20 presents an example of a conciliated model subset for our running example. Finally, in step sixth the engine writes a conciliation model conforming to the conciliation metamodel (CMM) in a well-formed Ecore file. At the end of this stage, and following the guidance of SBM, the engine was capable of gathering the information from all SPL domains in a single repository. The conciliated model contains the information for quality configuration analysis of the product under construction. Section 7 uses this model as input in order to support the SPL production plan assessment and a specific product configuration. 7. Product evaluation and filtering In the final stage of our proposal, we execute an evaluation algorithm over the conciliation model generated in the previous stage. The algorithm has three principal steps: the restriction validation, scenario evaluation, and reporting steps explained in the following sections. 7.1. The restriction validation step In the restriction validation step, the algorithm verifies whether, given the model’s AttachedProperties for each ArchitecturalElement, the possible Implementations can implement the element given its Restrictions. The engine uses two matching processes to find the correct evaluation concept. The engine first looks for all AttachedProperties in the element with the same Measure attribute values as the one in the Restriction ([name], [type], and [unit]). If the engine finds multiple concepts that satisfy the matching restrictions, it uses the property [eval] of the Measures as the differentiator attribute. This

551

attribute crosses the domains of the SPL, and the architect can use it to trace the possible required and offered quality properties of its elements. If the elements are comparable, this attribute has the same value. Subsequently, the algorithm scores the Measures of the required Restrictions, accounting for Measures in the AttachedProperty. As both concepts are comparable in this case, the scoring process evaluates how well the Implementation will perform, given its Restrictions, over the AttachedProperties of the implemented ArchitecturalElement. Let’s refer to the AttachedProperty measures as the expected measures, and to the Restriction measures as the given measures. The first thing for the scoring process is to find the middle points of the intervals in each of the given and expected measures. The interval of each measure is defined by its [max] and [min] attributes. The middle point or pivot of the interval will serve as the comparative point between both measures. The next step is to calculate the score of the given pivot in relation to the expected one. It means, how positively, or negatively far is the given pivot in relation with given one. To do so, the routine calculates this distance as a percentage. The routine is capable to calculate how larger or smaller is the given pivot from the expected pivot taking as the distance unit a percentage of the expected pivot. To illustrate the scoring process, imagine a ‘‘MainMemoryCapacity’’ quality attribute where expected pivot is 15 MB (middle point between 0 and 30 MB), and the given pivot of the ‘‘InstanceMemoryConsumption’’ measure in ‘‘PMSecureAES128’’ is 28 MB (middle point between 20, and 36 MB). Using the 100% of the expected pivot ‘‘MainMemoryCapacity’’ as the reference unit for the distance, the ‘‘PMSecureAES128’’ has a restriction score of +86.6%. It means, it is a +86.6% far of the expected pivot. (the +86.6% of 15 is 13, so the given measure exceeds in 13 measure units the expected pivot). This measure exposes the behavior of the implementation restrictions given the architectural element attached properties. In our algorithm, the higher scores are the worst and the lower ones are the best. We decided to take the expected pivot as a distance unit in order to get a normalized score for all the implementation evaluations. Using this strategy, if two implementations have the same restriction, they are compared with the same relative unit for the cohesion of the analysis.

7.2. The scenario evaluation step The second step of the algorithm evaluates the concrete component implementations accounting for the expected quality scenario measures declared in each sensitivity point. In this case, the evaluation process is similar to the one above. The scoring process is the same, but the comparative subjects are different. In the Dr.Chat Mobile example, the ‘‘SensitivityPoint persistenceSP1’’ has a Stimulus ‘‘StoreConsultation’’ with a Response concept ‘‘ResponseTime’’, and an expected Measure concept seconds. The algorithm scores all impacted artifacts (Implementations) according to their responses regarding a stimulus. Among the impacted artifacts of a Stimulus, the algorithm looks for the BehavioralProperties or Restrictions that have a comparable Measure to that of the stimulus. The algorithm then scores the artifact according to its given measure pivot. In this case, the expected pivot comes from the Measure interval in the Stimulus’ Response concept, and the given pivot, related to the Measure interval in the BehavioralProperty or Restriction concepts, comes from the Implementation. As in the previous step, the engine keeps this information as a scoring variable per Implementation, used in the final report. The ’’PMSecureAES128‘‘ in our example has a BehavioralProperty ‘‘ResponseTime’’ at the ‘‘StoreConsultationAES128’’ Capability. It has a measure comparable to the expected ‘‘ResponseTime’’ Stimulus’ Measure.

552


7.3. The reporting step

8.2. Experiment design

The algorithm presented here calculates the overall score of each possible implementation relation combination for the SPL. Based on the conceptual framework presented in Section 3, let’s define a set of all possible implementation relations (APIMs) as a conditional Cartesian product. In the formalization, the set A represents the architecture’s abstract components, and set C the concrete components of the SPL core asset base. The relation aðc:i; a:iÞ models the implementation of the abstract component a.i by the concrete component c.i, then the reporting step uses the APIM set defined as:

We designed one external configuration for a Dr.Chat Mobile application shown in Fig. 14. We also built a scenario suite with a set of expected quality scenarios for a product under construction. The experiment is based on two transactions to be performed with a Dr.Chat Mobile application. The transactions represent a user’s stimulus from an application functional perspective (e.g. execute a consultation transaction). The first transaction (TR1) requires the application user to create and submit a medical consultation. The following steps comprise the transaction. Once the user is logged into the system, the application must locate the user (using either the device’s sensors or manual selection from the index-based lists). The user must then describe his medical condition using text and attachments, including voice or video recording and pictures. Finally, the application sends the consultation to the medical server. The second transaction (TR2) requires the application user to retrieve a previously submitted medical consultation and check its attachments. In this transaction, the user first selects a medical consultation from the medical record stored in cellphone’s memory, then retrieves from the server its attachments including voice or video recordings and pictures. Our experimental design contains two main parts. The first part (XP1) focuses on testing the human analyst’s behavior, while the second (XP2) tests SPAF assistance capabilities. In the first part (XP1), the human analyst must manually select a set of concrete components from the components in the core assets base to instantiate product architecture. Given a scenario suite, the analyst must then find possible sets of components that can fulfil the responses according to the stimulus in the quality scenarios in the context of (TR1) and (TR2). To test the selected components, the human analyst produce a product production plan for each set of components, and use the SPL to derive the product. For each product the suite tests are executed and using the monitoring tools she collects the top-rated ones according to its functional and quality characteristics. In the second part (XP2), the human analyst must model the Dr.Chat Mobile SPL software architecture, variability, component catalog, and scenario suite using the proposed languages for each domain provided by SPAF. The tester must also model one sensitivity point for each scenario in the provided suite case. The human analyst must then run the SPAF tool and collect the top-rated components according to its functional and quality characteristics. The analyst uses the components proposed by SPAF to define the product production plan and then use the SPL to derive the products. A detailed description of the test procedure, subjects, objects, and recording mechanisms is presented below.

APIM ¼ ððc; aÞjc 2 C : â 2 A ^ aðc:i; a:iÞÞ The algorithm first finds all possible subsets in APIM that, throughout the implementation relations defined by the tuples (c, a), can realize each a 2 A in a one-to-one relation with a c 2 C. Given that aðc:i; a:iÞ is a C A multivalued function, a subset meeting this condition is a possible SPL configuration (in a configuration, each abstract component is realized by a single concrete component). The algorithm then adds the scores of each c.i in the configuration sets and assigns the resulting score as the overall configuration score or ratio. The ratios are the sums of all scores given to each realization relation in the configuration. In a complete report, the scores of all components, given their respective restrictions and quality scenario responses, count in the overall configurations ratio. If there is a CPU consumption scenario for the ‘‘LocalizationManager’’ abstract component implementations, the individual ‘‘GPSLocalization’’ and ‘‘WIFILocalization’’ concrete component scores affect the ratio of each configuration that includes those implementations. The final procedure in the reporting step is organizing possible SPL components in ascending order. The report is the final output of our evaluation process; even when the overall response is not an optimization analysis method, it supports the product builder during the definition of product production plans. With the report, the product builder can browse the SPL internal configurations that answer the external configuration of a product to derive. Moreover, it organizes and prioritizes the product line information and makes it treatable for human analysis.

8. Case study validation In this section, we present the experiments used to test our proposal’s effectiveness in the Dr.Chat Mobile SPL context. The experiment aims to study that using the SPAF framework, the SPL architect more precisely finds the internal possible components that meet the SPL production plan and fulfil a set of expected quality scenarios for a product to derive. In the experiment, selecting the concrete components to be used in a product production plan in an unassisted way is compared to the selection of components using SPAF.

8.1. Goals In this experiment, we test the effectiveness of an SPL product builder during the definition of a product production plan using the Dr.Chat Mobile SPL. We particularly compare the results obtained when selecting the components for an SPL production plan following the traditional manual process, with the results obtained when assisted by the SPAF tool. We do not compare the time taken by a human versus SPAF, but rather the quality of the SPL production plan, and a product specific configuration when SPAF is used.

8.2.1. Subjects The Dr.Chat Mobile generated application is deployed for the Android-Java programming language. Following an abstract factory pattern, the SPL uses the product production plan to configure the product in compilation time using a set of specific concrete components. Using this procedure, the analyst can first derive a Dr.Chat Mobile product, run it, and record its behavioral characteristics. Prior to the experiment, all components were tested independently. The tester was a software architect who was experienced in Android development, software architecture design, quality attribute requirements, and sensitivity point definition. We deployed the SPAF analyzer in a Windows 7 Machine (8 GB RAM, 250 GB DD, and a 2.5 GHz Core i5-2520 M CPU). The cellphone used to test different product configurations under the scenario suites was a Samsung Nexus S (1 GHz Samsung Hummingbird) running Android 2.3.3 (Gingerbread), with 512 MB of RAM and 16 GB iNAND of Storage (1 GB Internal and 15 GB External).

553


Fig. 14. Test selected features – test configuration for the Dr. Chat mobile SPL.

Table 1 Suite case SPL testing scenarios. ID

Source

Stimulus

Artifact

Environment

S1S1 S1S2 S1S3 S1S4 S1S5 S1S6 S1S7

User User User User User User User

TR1 TR1 TR1 TR1 TR2 TR2 TR2

Video manager Audio manager Image manager Persistence manager Video manager Image manager Persistence manager

Normal Normal Normal Normal Normal Normal Normal

operation operation operation operation operation operation operation

(single (single (single (single (single (single (single

user) user) user) user) user) user) user)

Response

Measure

Memory consumption AVG CPU consumption AVG Total storage consumption Total response time Total memory consumption AVG CPU consumption AVG Total response time

6230.000 KB 631% 63.000 KB 63 s 6235.000 KB 650% 66 s

Table 2 Dr. Chat Mobile external configurations realizing the variation points from the internal configuration (selected by a human analyst). E. Config.

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17

Dr. Chat Mobile variable abstract components Localization M.

Video M.

Audio M.

Image M.

Persistence M.

Zone index Zone index Zone index Country index Country index Country index World index World index World index WIFI WIFI WIFI WIFI WIFI GPS GPS GPS

MPEG4 MPEG4 MPEG4 MPEG4 MPEG4 H263 H263 H263 H263 H263 H263 H264 H264 H264 H264 MPEG4 H264

raw AMR raw AMR raw AMR raw AMR raw AMR 3GPP 3GPP 3GPP 3GPP 3GPP MPEG4 MPEG4 MPEG4 MPEG4 raw AMR MPEG4 MPEG4

VGA VGA VGA 1MP 1MP 1MP 2MP 2MP 2MP 3MP 3MP 5MP 5MP 5MP 5MP 5MP VGA

Blowfish32 Blowfish32 DES56 DES 56 AES128 AES128 TDES128 Blowfish169 AES192 AES192 TDES192 TDES192 AES256 Blowfish448 AES256 AES256 AES256

8.2.2. Objects The external configuration analyzed is a Dr.Chat Mobile application with the following features: Localization, Non-Secure Mobile Communication, Secure Persistence and Audio, Text, Video, and Image information input features. Fig. 14 portrays this configuration. The suite cases used during the experiment test four quality scenarios. While the first scenario concerns the average memory consumption when executing a transaction, the second one deals with the average CPU usage by the same functional execution. The third suite concerns the total storage resource consumption of the application, and the fourth scenario deals with the total response time of a transaction. Table 1 presents the quality scenarios associated with the experiment. 8.2.3. Instrumentation To collect performance data, we used the Alten TracingMemory application for Android. We selected this application for its freeware characteristics in the Android Marketplace. This tool can also export the measured metrics (resource consumption) in well

Table 3 External configuration subjects testing suite 1 fail counting. Conf.

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17

Scenario ID

Total fail count

S1

S2

S3

S4

S5

S6

S7

0 0 1 0 1 1 1 1 0 1 1 1 0 1 0 0 0

1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 0 0

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0

0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 1

1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0

0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 1 0

2 1 1 0 2 2 3 2 1 5 4 7 6 6 6 3 1

554


Fig. 15. TOP 25 optimal configurations – SPAF analyzer.

formed CVS files, which are browsable using a computer for analysis. For the time response scenarios, we developed a time-stamp log strategy where critical points in an application’s life cycle are time stamped and written in a plain text file. This plain text contains the time response measure for the specific transactions. As mentioned above, this strategy ignores the time consumed by the user when interacting with the system. It collects data about the holding time exposed by the application when executing a transaction. During the data collection procedure, the tester configured the Dr.Chat Mobile application according to the external configuration. The tester then installed the configured application into the subject’s cellphone and runs each test transaction sequentially. The tester recorded information about the resource consumption of the application (memory and CPU) in parallel. In the same manner, the experimental application writes a log with a time stamp related to transaction event occurrences. At the end of each execution, the tester extracted the information from the CVS files for transaction and/or configuration, reviews response time information, and manually calculates the cellphone’s storage resource usage. All attachments related to continuous event recording (including voice and video recording) last 15 s in all tests.

8.3. Execution 8.3.1. Execution – Part 1: XP1 Given an product configuration to fulfil, paper documentation about the SPLA, and the components in the core asset base, the tester selected 17 possible SPL set of concrete components to be used in the product production plan. Table 2 presents these configurations. Secondly, the analyst tested the scenario suite. For all the component sets found, the human analyst configured the product production plan, generates the application using the SPL, ran the experiments, and recorded the related performance items to verify each quality scenario in the suite. Table 3 exposes the fail count of each configuration related with the scenarios inside the suite presented on Table 1. In the Table 3, 0 specifies that the configuration have passed a scenario meeting the expected quality scenario measure. On the contrary, the number 1 expresses a failed execution where the application did not meet the expected measure for the stimulus. After performing the experiment’s first part (XP1), the human analyst found 17 possible component combinations that conform a set of viable internal configurations that can implement the var-


555

Fig. 16. Sensitivity point zoom – SPAF analyzer.

Table 4 Report summary – Dr. Chat Mobile. Technique Manually inspection SPAF assisted

Functional possible configurations

Configurations quality filtered

Filtering (%)

17

1

5.88

1620

543

33.33

iation points selected in the external configuration. After the test execution, the tester selected only one configuration (C4) as the one that met the expected quality responses communicated in the scenario suite. The human analyst took 1.3 h to select the 17 initial internal configurations and 10 h to execute the final quality filtering process. 8.3.2. Execution – Part 1: XP2 For experiment XP2, the architect introduced all experimental objects to the SPAF analyzer. The architecture was modeled using Archivol’s architectural package, the SPL core asset components were modeled through the CAMM, its variability using the simplified feature metamodel, and scenarios S1–S7 were modeled using the Archivol Quality package. The SBM modeled for the execution of the experiment in the SPAF analyzer is presented in Listing A.22. After the composition and evaluation procedures, the report presented in Figs. 15 and 16 was generated. The SPAF analyzer found 1620 possible internal configurations to answer the given external configuration. The report example portrays the top 25 rated configurations. Given that the SPAF analyzer implements a deterministic algorithm to find the cross product between model sets (with the model structural comparison support), the 1620 configurations are all the possible configurations that meet the given external configuration. The report in Fig. 16 shows the top-rated configurations after the quality evaluation. The first 543 reported configurations have a negative overall ratio. The report suggests that nearly 33.3% of the functional configurations are viable from a quality perspective. Moreover, the report shows that not even the top configuration

possesses all of the suite scenarios scored with a negative value, or zero. However, the top 18 rated have 6 quality scenario evaluations (S2, S3, S4, S5, S6, S7) with a negative score, and an S1 evaluation less than +2% of the score’s distance to the expected value. Fig. 16 illustrates the first report page generated. It shows details on the evaluation of a sensitivity point in the scenario S1 of the top-rated configuration. After the experiment’s second part (XP2), the architect, assisted by the SPAF analyzer, found 1620 possible component combinations (internal configurations) that can implement the variation points selected in the experiment’s external configuration. Posterior to the quality evaluation and reporting process, SPAF suggested to the architect that 543 configurations that met the overall expected quality responses of the test suite. The results on the report have been prioritized and can be browsed in order to explore the quality assessment summary. Nonetheless, the architect is advised to use the top-rated configuration as the main source of advice in the configuration process. Table 4 summarizes the experimental execution for both tested subjects (human analyst and SPAF engine). 8.3.3. Discussion Regarding the questions that motivated our experimental design and execution, we consider the following thoughts: RQ1: Can the implementation of a domain specific modeling strategy improve the representation of the information that constitutes the assets of a SPL? Yes, while implementing Sensitivity Point Analysis Framework and using the formal languages provided in our proposal, the engine automatically seeks a set of components that satisfies a set of requirements. In our implementation, the SPAF analyzer was 98% more effective than non-assisted selection in searching and property identification in SPL assets. Furthermore, once the SPL assets have been modeled using the languages provided, they can be queried, explored and validated using MDE tools such as QVT [27] and OCL [28]. RQ2: Can implementing a model analyzer improve a software architect’s decision making when selecting the most suitable combinations of reusable components on a SPL configuration time? Yes, the SPAF analyzer found all possible internal configurations

556


Fig. A.17. Example of Dr.Chat Mobile architectural view model subset using Archivol.

that, from a functional perspective, answer the experiment’s initial conditions. While the unassisted technique found 17 possible configurations, the SPAF analyzer found 1620. RQ3: Does the mentioned analyzer improve component selection with a set of quality requirements as input? Yes, the SPAF analyzer found 543 possible configurations that fulfil both the functional and expected quality dimensions of a to derive. The SPAF analyzer improved 99.7% of the unassisted analysis selection capabilities when selecting the most suitable set for a product’s functional and behavioral configuration. In the first run, while the non-supported analyst finished in 11 h (experiment XP1), the SPAF-assisted analyst took nearly 38.1 h in experiment XP2. The time difference results from the time required to model the assets using the provided metamodels and domain-specific languages. Despite the time of the first run, the supported analysis will take less than 11 h to execute the SPAF analyzer if the assets have been already modeled. This information has been collected in experiments configuring more products medical tele-consultation products for the FSFB using the SPL once the assets have been modeled by the languages provided by SPAF. 9. Threats to validity In this section, we analyse the conceptual, internal, and external factors that might threat the validity of the results obtained with the sensitivity point analysis framework. We based the presentation of the such validity factors following the guidelines and categories found in [29] and exploratory questions used in [30].

9.1. Construct validity (Do we measure what was intended?) We have compared the effectiveness measure of a product configuration within a SPL supported by SPAF in terms of (i) discovery completeness of the configuration functional possibilities, and (ii) accuracy of the quality based configuration filtering. This quality driven approach for the selection of reusable artifacts has been studied by several authors. However, we have used the terminology and practices presented by the Software Engineering Institute in [25,13,6]. The domain specific modeling and, in general, model driven engineering approaches that we have used in order to model the software artifacts under study suffer from restrictions such as traceability between models, and human-dependent information accuracy. Nonetheless, the information has been introduced by experts and obtained by specialized tools that have been independently tested, and which performance metrics have been used by the community. 9.2. Internal validity (Are there unknown factors which might affect the outcome of the experiments?) The main internal threat to validity of the study is constituted by the limited number of human testers that were exposed to a product configuration problem without the SPAF assistance. Similarly to the external threat factors, training and getting a reliable SPL configuration manager or architect requires large amounts of time. However, our results overcome by far the human ability on the treatment of the combinatory and contextual


557

Fig. A.18. Example of Dr.Chat Mobile architectural concrete core asset model subset using CAMM.

problem that SPAF is addressing. Consequently, given the complexity of the computational task under observation, we do not expect any human tester to provide more accurate results that the ones provided by our framework. The amount of time required to perform the same study in order to get an statistically significant recall for the precision of the quality based configuration filtering is rather impractical. However, this study is a first step towards the validation of this metric in a real, but controlled SPL experimental environment.

in the context of performance-specific quality scenario suites. Even though the provided languages, and evaluation engines are flexible to model, and measure any type of quantitative information, we cannot claim that our findings can be held true for considerably larger SPLs, or evaluations performed out of the space of the performance quality scenario assessment.

9.3. External validity

In this section we present work related to our problem statement and solution strategy. We identified numerous trends with different advantages and limitations pursuing the problem stated in this article. The goal based model proposed in [31,32] recognizes the need to evaluate the variability impact instead of the quality that a product may hold inside a SPL. This proposal models the functional aspects of products inside an SPL as a dimension correlated with its quality (non-functional) requirements. The authors propose a hierarchical tree model for each of the domains (functional and non-functional) as a modeling technique, and qualitatively present a correlation scheme that joins both models. The quality model is a soft-goal model, and its functional homologous as functional goal model. The correlation links between two models represent the influences that the functional model exercises over the soft-goal

(To what extend is it possible to generalize the findings?) There are two main internal threats to validity in our study. The first one is defined by the size of the software product line that we have used as primary case study. Even though it is a software product line used to produce real products by the FMSB in the health sector, it counts with nine abstract architectural components, and a concrete component catalog with 64 artifacts only. In this context, other real SPLs could surpass this number considerably. However, most of the available SPLs are not well structured and do not contain enough information for the utilization of informationexhaustive modeling strategies. We consider that our case study shows a real SPL in an important context that represent the small and medium size SPLs. As a second threat, we relied on a case study

10. Related work

558


Fig. A.19. Example of Dr.Chat Mobile variability model subset using FMM.

Fig. A.20. Example of a conciliated model subset using CMM.


559

Listing A.22. Example of Sensitivity Definition Model using the SPDL.

model. The main problem in this technique is the low level of granularity in the correlation model. Even when the qualitative measures are extendable to all possible quality attribute perspectives, they present little comparative reference between them. The correlations lead the architect to consider the positive or negative impact a functional goal has over a quality soft-goal node. The comparative analysis is small given the differentiation characteristics of the quality attributes in a real case study. Secondly, given tangled models, their maintenance and co-evolution result in a difficult process. Thirdly, the combinatorial analysis poorly defines the possibilities that can satisfy a product’s configuration. The

authors present an interesting prioritization scheme, excluded by our proposal: this uses a prioritization technique for the soft-goals to give weight to the scoring decision support. With this strategy, the authors provide a tool to prioritize the more important attributes for a specific product under construction. The proposal’s evaluation involves neither the possible internal product line configurations nor its architecture. Finally, this proposal’s tools are embedded in Excel and Visual Basic extensions. We define our proposal as well formed languages over well recognized modeling frameworks that allow extension and standard use close to the end user’s domain.

560


The soft goal interdependency graph proposed in [33] addresses a similar approximation to the Goal Based Model in [32]. The proposals differ in their utilization of a well formed language based on feature model abstract syntax to express the functional product line goals. As the authors of [34] express, this methodology assigns qualitative measures to the functional goals’ impacts over the softnon-functional ones. Despite introducing a well-formed expression mechanism for the functional product line goals, the authors do not provide an analysis scheme to deduce the implicit information inside the models. The main limitation of the proposal is thus that even when the models serve as information repositories, there is no conclusion of the relations expressed on them. Given the number of possibilities for goal selection in the functional model, manual analysis makes quality verification non-viable and error-prone. Additionally, the authors do not involve SPL architecture in the modeling process. The extended feature model with constraint satisfaction problem analysis proposed in [35,36] presents a strategy for modeling the SPL scope using Czarnecki’s feature metamodel [12]. The authors proposed a metamodel extension to attach different quality properties to the feature nodes. The attached feature node properties can be at discrete or continuous intervals and, in a multi-level approach (over the tree), can reference other properties as functions. This characteristic allows expressing, with high granularity levels, property measures. The proposal includes a Constraint Satisfaction Problem (CSP) approach [37] to consider each interval on a finite domain. The automatic reasoner limits the values simultaneously taken by the feature nodes to filter those that can accomplish the expected restrictions of a product to derive. This proposal’s main limitation is that it does not include internal product line variability characteristics. Given that the attached properties are directly related to the feature nodes, analyzing different internal configurations is impossible (only external are possible). The proposal does not generally allow a variation point to be realized by multiple SPL artifacts. A possible solution is modeling a tree for each abstraction level, (from the external to internal variabilities). However, methods to relate each tree and its extensions as a unified model across SPL abstractions levels to execute a CSP evaluation strategy remain unclear. As in the previous proposal, authors in [35] do not include the SPLA in its contextual analysis; the information is provided explicitly in the proposed extension concepts. Another extended feature model proposal is presented in [38]. It does not address the problem as a CSP, instead builds two evaluation and scoring models. The proposal uses an extended feature model in which a generic high level evaluation model is first exposed. Using qualitative measures, the authors model properties attached to the variable SPL function features. The authors then assign a weight to each attached property: for example, a property that was scored ‘‘++’’ on a high level, can be scored as ‘‘++ Numeric Value’’ on a lower level. This weight is calculated independently given the available SPL artifacts to realize a variation point. The evaluation used to examine model information selects a configuration with the best scoring features. This proposal does not include three principal factors that we consider critical. It first ignores the low variability level. As with other proposals, this one assumes that only one SPL artifact can realize a variation point in a system to build. Secondly, for the same reason, it ignores the combinatory analysis of the assemblage possibilities. The scoring process selects only one viable configuration. However, in a real SPL, there can be multiple configurations that meet the expected behavioral properties for the product under construction. Finally, this proposal does not include SPL architecture in its evaluation procedure. The architect must indicate all information as numeric values and functions in the properties attached to the variation points; if the product line architecture changes, all values and formulas have to be recalculated.

Lee and Kang present in [11] a domain knowledge model to support the configuration process of a product using a SPL combining its functional elements, non-functional features, and usage contexts. The authors approach the variability modeling of a product line taking into account that similar functionalities are implemented differently depending on particular product contexts. These context are conformed by the user, the physical deployment environment of the product under derivation, and its business and social and operating environments. Authors propose the modeling of the different context through feature models during the domain engineering process of the product line. Similarly, non-functional features like quality attributes are modeled orthogonally using feature models as well. The proposal main contribution is a knowledge model where the orthogonal models of quality attributes and product features are mapped according usage context information. The intersection of the models will expose the assessment of particular functional features regarding its quality attributes in a given usage context. This assessment is a coarse-grained representation of the models intersection with ‘‘++’’, ‘‘+’’ or ‘‘’’ discrete values depending on how favorable is to use a particular feature implementation from a quality attribute perspective in a given product context. This implementation extends our proposal providing additional contexts where the quality attributes can be affected and have to be evaluated. However, similarly to the other existing proposals, the scoring mechanism is not supported by an automatic tool. Additionally, it evaluates each possible feature selection assessment separately with coarse-grained states. Our proposal instead summarizes the configuration scores in a digested, numeric, and automatic way. Moreover, our tool involves the architecture of the software product line as a fundamental asset of the configuration process by introducing sensitivity points where a feature can behave differently depending on the architectural structures and its relationships with the SPL assets.

11. Conclusions In this section we discuss the limitations and conclusions of our research. We have verified that the total number of possible configurations in a product line reaches hundreds of thousands. Through our case study, we validated that the evaluation of all the possible SPL internal configurations to derive a product is a process which cannot be treated by humans in a practical way. Hence, our research sensibly seeks a human-treatable simplification using a set of computational tools. In our experiments, we have shown that not all possible component combinations can both implement a configuration and simultaneously satisfy a set of quality scenarios for a product to configure and derive. These results and our experiments motivate our research work. We have found that the number of variables is enormous, and the actual internal configurations show variable responses to the same external configuration. Our composition and analysis capabilities relate to verifying the expected and given values of a set of concrete artifacts that can realize a specific configuration and implement the software architecture of a product. We can find relations between expected stimulus metrics and those presented by system artifacts when assembling a product. Our analysis mechanism does not use an optimization scheme, but provides a valuable decision support tool to configure a product within the product line. The best combination of artifacts to configure a product is not always the best scoring. We consider the bad scored configuration to be as valuable as the best one. With this information, the architect can modify its architecture or deprecate artifacts from the SPL catalog to get higher quality products. We did not want to provide a unique answer to the configuration, but a set of viable ones with enriched


information to support the architect’s decision making. Given the amount of SPL information captured in a sensitivity point model, we believe that future analysis engines can generate reports about SPL characteristics. An engine can create a report presenting nonviable assets. The highlighted assets in the SPL always make a stimulus–response analysis fail in a given configuration. With this report, the architect could execute elicitation processes to replace low quality artifacts in an SPL artifact catalog. Through our Dr.Chat Mobile experiment, we have realized that our approach can filter nearly 85% of the product line configurations that cannot fulfil the expected quality attributes for a derived product. It discards these configurations according to restriction analysis and given response measures for a set of specific stimuli. The number of possible configurations can reach hundreds of thousands, or even millions. When the number has that order of magnitude, the quality fulfilment ratio organizes 15% of non-filtered configurations to allow the architect to manually and practically analyze them. We also do not only expose a configuration by itself, but our report also includes the scores’ origins and which are the product’s evaluated quality scenarios. We have provided a source of browsable information as decision making support. As stated in the previous sections, we have identified multiple extension points to our proposal. Our current research does not execute constraint verification processes over the SPL domain models. In the architecture modeling process, the elements inside a view family should contain architectural elements according to their view family type. In our implementation, this modeling behavior is not controlled. When modeling the SPL concrete assets with the Concrete Asset Metamodel (CAMM), the modeler must manually introduce the metrics of a specific restriction or behavioral property related to a contract implementation. However, future work could address the automatic generation of these metrics. The algorithms used to analyze and process the conciliation model could be optimized to improve the resources and time required in our solution. Additionally, the automatic verification of inconsistencies in the models used in our solution needs to be manually validated. At this point, our strategy depends on the correct definition of these models. For our proposal to evolve, continuing to use our metamodels to express the SPL domains from different contexts and modeling perspectives is fundamental. We have tested our proposal in two different case studies. The first is the mobile medical teleconsultation SPL, as presented in this article; the second is a Web Shop case study presented in [39]. Given this proposal’s possible extension points, extra efforts in SPL development are necessary because, in our approach, the architect must learn and use the designed DSL and modeling techniques. Acknowledgments We would like to thank the capacitation division of the Fundacion SantaFe de Bogota (FSFB) for the support, compromise and help offered during these years. We would like to thank also the members of the Model-Oriented Software Architectures (MooSAS) team. Appendix A. Dr.Chat SPL relevant models See Figs. A.17–A.20 and Listing A.22. References [1] K. Czarnecki, M. Antkiewicz, C.H.P. Kim, S. Lau, K. Pietroszek, Model-driven software product lines, in: Companion to the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, ACM, New York, NY, USA, 2005, pp. 126–127.

561

[2] J. Bézivin, In search of a basic principle for model driven engineering, The European Journal for the Informatics Professional 5 (2004) 21–24. [3] K. Pohl, G. Böckle, F. Linden, Software Product Line Engineering: Foundations, Principles and Techniques, Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005. [4] R. France, B. Rumpe, Model-driven development of complex software: a research roadmap, in: FOSE ’07: 2007 Future of Software Engineering, IEEE Computer Society, Washington, DC, États-Unis, 2007, pp. 37–54. [5] M. Sinnema, S. Deelstra, J. Nijhuis, J. Bosch, Covamof: a framework for modeling variability in software product families, in: R. Nord (Ed.), Software Product Lines, Lecture Notes in Computer Science, vol. 3154, Springer, Berlin, Heidelberg, 2004, pp. 197–213. [6] P.C. Clements, L. Northrop, Software Product Lines: Practices and Patterns, SEI Series in Software Engineering, Addison-Wesley, 2001. [7] L.M. Northrop, P.C. Clements, F. Bachmann, J. Bergey, G. Chastek, S. Cohen, P. Donohoe, L. Jones, R. Krut, R. Little, J. McGregor, L. O’Brien, A Framework for Software Product Line Practice, Version 5.0, 2009. [8] J. Kuusela, J. Savolainen, Requirements engineering for product families, in: Proceedings of the 22nd International Conference on Software Engineering, ICSE ’00, ACM, New York, NY, USA, 2000, pp. 61–69. [9] H. Schirmeier, O. Spinczyk, Challenges in software product line composition, in: System Sciences, 2009. HICSS ’09. 42nd Hawaii International Conference on, pp. 1–7. [10] H. Zhang, S. Jarzabek, B. Yang, Quality prediction and assessment for product lines, in: J. Eder, M. Missikoff (Eds.), Advanced Information Systems Engineering, Lecture Notes in Computer Science, vol. 2681, Springer, Berlin, Heidelberg, 2003, p. 1031. [11] K. Lee, K. Kang, Usage context as key driver for feature selection software product lines: going beyond, in: Lecture Notes in Computer Science, vol. 6287, Springer, Berlin, Heidelberg, pp. 32–46. [12] K. Czarnecki, S. Helsen, U. Eisenecker, Staged configuration using feature models, in: Software Product Lines, 2004, pp. 162–164. [13] L. Bass, P. Clements, R. Kazman, Software Architecture in Practice, SEI Series in Software Engineering, Addison-Wesley, 2003. [14] T. Stahl, M. Volter, Model-Driven Software Development: Technology, Engineering, Management, John Wiley, Chichester, England; Hoboken, NJ, 2006. [15] H. Arboleda, R. Casallas, J.-C. Royer, Using transformation-aspects in modeldriven software product lines, in: Workshop Proceedings ADI08, 2008, pp. 46– 56. [16] S. Kent, Model Driven Engineering, in: IFM ’02, Springer-Verlag, London, UK, 2002. [17] D.C. Schmidt, Guest editor’s introduction: model-driven engineering, Computer 39 (2006) 25–31. [18] D. Gasevic, D. Djuric, V. Devedzic, Model driven engineering, in: Model Driven (Ed.), Engineering and Ontology Development, Springer, Berlin, Heidelberg, 2009, pp. 125–155. [19] L. Zhu, A. Aurum, I. Gorton, R. Jeffery, Tradeoff and sensitivity analysis in software architecture evaluation using analytic hierarchy process, Software Quality Journal 13 (2005) 357–375. 984BC. [20] N. Rozanski, E. Woods, Software Systems Architecture: Working with Stakeholders Using Viewpoints and Perspectives, Pearson Education, 2011. [21] Y. Peña, D. Correal, T. Hernandez, Reusing legacy systems in a serviceoriented architecture: a model-based analysis, in: Proceedings of the 2010 International Conference on Advances in Conceptual Modeling: Applications and Challenges, ER’10, Springer-Verlag, Berlin, Heidelberg, 2010, pp. 86–95. [22] D. Correal, Model Oriented Software Architectures, 2011. [23] J. Bayer, S. Gerard, Haugen, J. Mansell, B. Mller-Pedersen, J. Oldevik, P. Tessier, J.-P. Thibault, T. Widen, Consolidated product line variability modeling, in: Software Product Lines, Springer, Berlin, Heidelberg, 2006, pp. 195–241. [24] K. Kang, S. Cohen, J. Hess, W. Nowak, S. Peterson, Feature-Oriented Domain Analysis (FODA) Feasibility Study, 1990. [25] L.J. Bass, M. Klein, F. Bachmann, Quality attribute design primitives and the attribute driven design method, in: Revised Papers from the 4th International Workshop on Software Product-Family Engineering, PFE ’01, Springer-Verlag, London, UK, 2002, pp. 169–186. [26] V. Guana, Sensitivity Point Analysis Framework, 2011. [27] OMG, Request for Proposal: MOF 2.0 Query/Views/Transformations RFP, Technical Report ad/2002-04-10, Object Management Group, 2002. [28] J. Warmer, A. Kleppe, The Object Constraint Language: Precise Modeling with UML, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1999. [29] R. Yin, Case Study Research: Design and Methods, third ed., Sage Publications, 2003. [30] Q. Soetens, S. Demeyer, Studying the effect of refactorings: a complexity metrics perspective, in: Proceedings QUATIC, vol. 10, 2010. [31] B. Gonzalez-Baixauli, J.C.S. d. P. Leite, J. Mylopoulos, Visual variability analysis for goal models, in: Proceedings of the Requirements Engineering Conference, 12th IEEE International, RE ’04, IEEE Computer Society, Washington, DC, USA, 2004, pp. 198–207. [32] B. Gonzalez-Baixauli, M.A. Laguna, Y. Crespo, Product line requirements based on goals, features and use cases, in: International Workshop on Requirements Reuse in System Family Engineering (IWREQFAM), pp. 4–7. [33] S. Jarzabek, B. Yang, S. Yoeun, Addressing quality attributes in domain analysis for product lines, IEEE Proceedings-Software 153 (2006) 61–73.

562


[34] L. Etxeberria, G. Sagardui, L. Belategi, Modelling variation in quality attributes, in: First International Workshop on Variability of Software-Intensive Systems (VaMos’07), volume Lero Technical Report, Limerick, Ireland, pp. 51–59. [35] D. Benavides, S. Segura, P. Trinidad, A. Ruiz-Cort’es, A first step towards a framework for the automated analysis of feature models, Technical Report, 2006. [36] D. Benavides, S. Segura, A. Ruiz-Cortés, Automated analysis of feature models 20 years later: a literature review, Information Systems 35 (2010) 615–636.

[37] E. Tsang, Foundations of Constraint Satisfaction, Academic Press, 1995. [38] L. Etxeberria, G. Sagardui, Variability driven quality evaluation in software product lines, in: Software Product Line Conference, 2008. SPLC ’08. 12th, International, pp. 243–252. [39] V. Guana, D. Correal, Variability quality evaluation on component-based software product lines, in: Proceedings of the 15th International Software Product Line Conference, SPLC’11, vol. 2, ACM, New York, NY, USA, 2011, pp. 19:1–19:8.