Maintenance-oriented selection of software ... - Semantic Scholar

Maintenance-oriented selection of software components Pasquale Ardimento, Alessandro Bianchi, Giuseppe Visaggio Dept of Informatics - University of Bari – Via Orabona 4, 70126 Bari - Italy; RCOST - Bari {ardimento, bianchi, visaggio}@di.uniba.it Abstract Component-based software engineering is a new, promising, and rapidly growing discipline in both academia and industry. However, maintaining component-based systems (CBSs) introduces new issues: the choice of the components requires identifying a set of parameters that characterize them, in order to select the appropriate ones for a specific software system. In our research we propose a characterization of components aimed at foreseeing the maintenance effort of the CBS. In this paper we perform an empirical study in the context of three industrial software projects to assess these parameters. Our experience suggests a number of components characteristics, which can be useful for the purpose above. Moreover, the study produced some lessons learned, useful for building software applications easy to maintain. The results show that the lessons learned could be generalized, although further empirical studies are required.

1. Introduction Component-Based Software Engineering (CBSE) is a rising discipline in recent years which faces the problem of reducing complexity and cost of software processes. The methods and technologies it provides stress the reuse of existing software components in order to build new systems. Despite its good effects on the overall software process, it introduces, among the others, the problem of choosing the components, which are the most appropriate for the target system. This choice impacts the development of the Component-Based Systems (CBS). However the problem of correct choice is emphasized during maintenance. In fact, maintenance is a critical activity, which consumes the majority of the effort spent within the lifetime of a software system, when the system is developed according to traditional methods and techniques. When adopting CBSE techniques, maintainability becomes even more critical. In fact, many problems can derive from the different evolution of the CBS and the components integrated in it. In most cases those evolutions are asynchronous, in the sense that - the CBS needs some specific evolution of components, but they do not evolve with the required scheduling, or - the components evolve even if their evolution is not required by the CBS.

Therefore, it is necessary to properly manage possible redundancy in the capabilities provided by the CBS, which can come from more than one component. Secondly, CBSs requires a deep knowledge of all the aspects of the involved components. Often, developers deepen into only some issues of the involved components, but during maintenance new issues can rise. Therefore, the choice of the component to integrate consider the capability to access knowledge of all aspects. In order to face these problems, the authors are performing empirical studies, aimed at identifying the most significant or relevant characteristics, which can help software engineers in selecting the components to be integrated in a target CBS. More precisely, in [1] a preliminary set of components characteristics have been identified according to authors’ experience. The goal of [1] was to select characteristics, which can be easily collected, in order to choose the most adequate components. We are now interested in evaluating the effectiveness of the identified set with respect to maintainability quality factor. For sake of clarity, in this paper we do not introduce new maintainability measures. Instead, we want understand (through maintainability measures) whether CBS maintainability can be foreseen using the characteristics of the involved components. The obtained results allow to learn some lessons, which can be useful in choosing components. The motivation of this empirical study derives from the need to provide support to the managers who have to decide the most appropriate component to include in CBSs. This support can help in reducing the risks during maintenance. To this end we analyzed maintenance data of three industrial CBSs. The components integrated within each of them were selected making use of specific criteria, depending on both the features of the organizations and of the systems to develop. In this paper by components we mean all the software systems existing a priori, available to the general public, integrated within a target system, and can be acquired in any way. This definition designates a large variety of software systems, including: legacy systems, free software, COTS (Commercial Off The Shelf) products,. According to the definition provided by Software Engineering Institute (SEI) in [2], these are software components that can be bought, leased or licensed. This

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

definition is useful in that, according to authors’ experience, it is what occurs in many real cases, in which organizations integrate legacy systems, COTS products and free software within new systems, in order to enlarge the set of provided services. The remainder of the paper is organized as follows: section 2 provides a brief overview of the main research activities presented in literature dealing with the same topics; section 3 presents the empirical study we executed, the results are presented in section 4, and discussed in section 5; section 6 draws the conclusions.

2. Related works In literature many authors acknowledge the need to adequately select the proper components to be integrated in a CBS, even though they focalize the analysis on different viewpoints. For example, Lawlis et al. [3] analyzed the difficulty to identify the most appropriate components for specific requirements; Basili and Boehm defined a “TOP 10 List” [4] which represents different research challenges for “enhancing our empirical understanding of component-based systems”. More specifically Hypothesis 9 states: “personnel capability and experience remain the dominant factors influencing component-based systems development productivity”. It refers to difficulty in learning the components features for the development of a software system. Moreover, Boehm and Abts in [5] identified the difficulty to integrate components within the CBS. The problem of selecting the proper components is also faced by a number of international research projects. For example, decomposing the general characteristics of components into a set of quality attributes has been exploited by [6], in the PECA project [7], and in the PORE project [8]. The concept key of components evaluation is to use quantitative attributes to measure the components conformance with a set of requirements [9]. These approaches differ in attributes definition and their reuse in different projects. For example, [10], [11], [12], and [13] propose to redefine attributes based on software system requirements. On the other hand, approaches like [14], [15] and [16] define a set of generic attributes (often inspired by the ISO 9126 standard [17]) that can be reused across projects and possibly organizations. All these studies mainly take into account the problems related to the development of the CBS, and to our knowledge, no studies have been carried out with respect to maintainability quality factor.

3. Empirical Study Our research can be characterized as an explorative study on data of three industrial projects and, according to [18], it has been executed with the following phases:

preparation, data collection and data analysis. The three projects have been developed through a CBSE approach. During preparation phase the goal has been defined as: - analyze the maintenance process of a CBS, with the aim to evaluate the impact of components quality factors on maintainability, from users’ perspective; - define the measures, which can be collected from component documentation or provided by managers. During data collection phase, the history of all projects has been reconstructed. This allowed us to make conjectures about the components characteristics. Data have been collected from both available documentation, and through interviews to the project managers. Finally, the collected data have been analyzed.

3.1. Projects characterization The three industrial projects analyzed in our research are named SICOD, DiPNET, and Portal. They present the features of enterprise applications, such as: an extended purpose; a high number of users (note that University of Bari has about 70000 students and it engages about 4000 people); a required performance adequate to the application usage. For sake of comprehension, appendix reports on the functionalities provided by each system. Moreover it is worth noting that the effort for building is not correlated to the extension of the goal of the system. In fact, thanks to the CBSE approach large CBSs can be built with relatively low effort. SICOD (Sistema Informativo per la COnvergenza con i Destinatari, i.e. Information System for Stakeholders Satisfaction) is a CBS supporting the qualification of the education and scientific services offered by the University of Bari. It has been developed within the Software Engineering Research Laboratory (SER-Lab) of the same university, by two software engineers with the “Laurea” degree in Informatics: with a 10 and 3 year experience respectively. The functionalities offered by SICOD are summarized in table A1 in the appendix, together with the components integrated in it and the cross reference functionality components. The components integrated in SICOD are: - Oracle 9i [19], as DBMS; integrated in SICOD through a white box technique - Oracle Internet Application Server 9i [19], as application server; integrated through a protocol; - Oracle Report Builder [19], and Crystal Report [23] for developing reports; both integrated through a proprietary protocol; - Applix iEnterprise [20], for managing the relationship to the stakeholders; integrated through a proprietary protocol; - Plumtree Corporate Portal 4.5 [21, 22] for the web portal, integrated though a wrapper.

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

Note that, the duplication of components for report management is due to integration requirements: reports concerning data management functionality are managed by Oracle Report Builder, reports concerning the Applix iEnterprise functionality are managed by Crystal Report. The Italian company Network Services developed the DiPNET (DIstributed Production NETwork services) software system, i.e., a web portal supporting different companies cooperating for the development of a project. It has been developed by two software engineers with the “Laurea” degree in Informatics and a 2 years experience. Table A2 in appendix summarizes the functionalities of DiPNET, together with components integrated in it, and the cross-reference functionality – components. The components integrated in DiPNET are: - SQL Server 7 [24] as the DBMS; it has been integrated in DiPNET through a white box technique; - SNITZ Forum [25], freeware for managing Internet forum; integrated by a white box technique; - DecisionScript, [26] application server developed by Vanguard Software for browsing knowledge base; integrated through TCP/IP; - DatePicker [27], freeware for calendar management; integrated through a white box technique; - MS ADO, COM objects for managing access to database, developed by Microsoft [24]; integrated through a proprietary protocol; - Internet Information Services (IIS), as the web servers, developed by Microsoft [24]; integrated through a proprietary protocol; - MS Index Server (IIXSO) [24 ]as a full-text indexing and search engine for (IIS); integrated through a proprietary protocol; - MS Collaboration Data Objects for NT (CDONTS) as an interface towards an SMTP server, used as a tool for sending e-mails [24]; integrated through a proprietary protocol. Portal is the Web-Portal of the University of Bari. It provides an unique access point to a number of services and information the university offers to its stakeholders. Portal has been developed within the SER-Lab by two developers, both with an experience of 1 year. It includes: - SICOD, i.e., the legacy system of SER-Lab, described above; integrated through SOAP; - SIANAR, a legacy system of SER_Lab providing the register of all the research activities carried out within the University; also integrated through SOAP; - Questionnaire, a legacy system of SER-Lab, manages (i.e., design, publishing, distribution and analysis) the questionnaires for evaluating services provided by the university; integrated through a white box technique; - Oracle Portal, providing interactive on-line services; and integrated by a proprietary protocol; - Decision Script [26], described above.

Note that the legacy systems SICOD, SIANAR and Questionnaire were developed independently before the need to use them into Portal. On the other hand, Oracle Portal and Decision Script have been bought from outside. Table A3 in Appendix summarizes the components integrated in Portal and cross refers each of them with respect to the functionality offered by the CBS. In the three projects, the involved people choose the components to include in the CBS, they integrated them and tested the resulting system, and finally they executed the maintenance activities. The changes concerned corrective, adaptive and perfective maintenance. Finally, in all cases, the authors of this paper managed, coordinated and supported the working teams.

3.2. Components characteristics In order to carry out this study, we identified a set of general-purpose characteristics of components, including: Adequateness of the component with respect to the target system; Costs required for training people in using the component; Familiarity of the working team with the component; Level of the support provided by the component’s vendor. According to our experience, these characteristics should act as indicators for choosing the proper components to be included into the target systems. The previous characteristics have been detailed in some metrics, collectable by each component. In the empirical study, metrics collection used available documentation or data provided by people involved in maintenance activities. Let’s briefly describe the measures of each characteristic; the complete list and details are in [1]. Adequateness In the context of this research, “Functionality” is the “capability” in the requirements specification of the CBS, therefore, it is expressed at the same abstraction level. Adequateness is detailed into the following measures: Functional Coverage, which represents the percentage of the CBS functionalities provided by the component. It is measured on a ratio scale. The value it assumes for the i-th component, Fun_Covi, is

Fun _ Covi =

Funi , where Funi is the number TotFunCBS

of functionalities provided to the CBS by the i-th component and TotFunCBS is the total number of functionalities the CBS provides; these can either be provided by components integrated within the CBS or developed from scratch. Let Total Function Coverage of a CBS be the sum of the Functional Coverage of all the components included in n

the CBS, i.e.,

TotFunCov = ∑ Fun _ Covi . Then, i =1

TotFunCov expresses the percentage of capability of each

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

component in supporting the overall functionality required by the CBS. Note that TotFunCov can be lower, equal to or greater than 100%. When it is: - lower than 100%, the components in the CBS do not cover all the functionalities offered by the CBS, and therefore some CBS functionalities have been developed from scratch; this case never presented in the set of industrial projects we analyzed. - equals to 100%, all the functionalities are covered by the set of components in the CBS, and the developers had only to glue them, without developing any functionality; this happened for DiPNET project. - greater than 100%, some functionalities offered by the CBS are provided by more than one component, so two or more components cover the same functionality. The latter is the case of both SICOD and Portal: SICOD integrates two components (i.e., Oracle Report Builder and Crystal Report) for the functionality concerning report management; Portal integrates two components (i.e., SICOD and SIANAR) for data management and three (SICOD, SIANAR and Oracle Portal) for report management. Therefore, in SICOD TotFunCov is 1,22 and in Portal it is 1,376. The third measure detailing the Adequateness characteristic is Compliance. It expresses the percentage of the CBS functionality provided by the component. For each component it indicates the usage within the CBS, as the amount of component functionality employed in the CBS. It is measured on a ratio scale and the value it assumes for the i-th component is

Compl i =

Funi , where Funi is the same as TotFunCOMP i

before, and TotFunCOMPi is the total number of functionality the i-th component makes publicly available. Therefore, Compliance expresses the effective usage of each component within the CBS. Note that, if a component Ci makes n functionalities publicly available, the number of functionality used within the CBS is m, where m ≤ n. Compliance values for each component can range between 0 and 1. Low values of compliance indicate that the component is poorly used within the CBS, and high values imply an intensive usage of the component. Training Cost. It is expressed by the Training Time (in the following TT), i.e., the working time spent for training people involved in the integration of components. People are trained through courses carried out by the vendor of the component. It is measured on a ratio scale, which indicates the number of working days the training courses lasted, where each working day is 8 hours long. Familiarity. The Familiarity measure expresses the understanding about a component, which has been acquired by software engineers in their previous

experiences. It is measured on an ordinal scale, ranging from Low, to Medium, to High. Support. Support measure expresses the level of support provided by the vendor (through documents, web pages, discussion groups, call service…). It is measured on an ordinal scale, with values Low, Medium, and High.

3.3. Quality Factor In this paper we focus on the maintenance quality factor, expressed through the “Mean Maintenance Effort” (MME) measure. For each maintenance request in the CBS, the maintainer design the changes needed in each component. Each change is then executed customizing the component, or requiring an explicit change to the component’s provider, or specializing the component with a second component or with an ad hoc program. MMEj represents the effort spent to satisfy maintenance requests r

in the j-th component of the CBS:

MME j =

∑ ME i , j

i =1

n

where MEi,j is the effort spent for satisfying the i-th maintenance request of the j-th component; r is the total number of maintenance requests; n is the total number of components in the CBS. MEi,j does not consider the possible effort, which has been spent by the provider for maintaining the component, but only considers the effort spent by the maintainer of the CBS. It is measured in persons days, where each working day is 8 hours long.

3.4. Normalization The set of components considered in this study is almost heterogeneous: in general, the values assumed by the measures MME and Training Time for a specific component are very different from the values the same measures assume in other components. Therefore, the values of those measures need to be normalized through the normalization factor NF for both measures: n

NF = ∑ V j , where Vj is the value the measure (MME, j =1

or TT) assumes for the j-th component, and n is the total number of components integrated within the CBS. Therefore, the normalized value of MMEj is

NMME j =

MME j n

,

and the normalized value of TTj

∑ MME j

j =1

is

NTT j =

TT j n

. On the other hand, normalization

∑ TT j j =1

of the remaining ratio measures isn’t necessary, in that the normalization factor is included in their definitions.

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

Table 1. Collected values about components integrated into the three CBSs CBS

Component

Oracle Oracle Application Server Oracle Report Builder SICOD Crystal Report Applix Plumtree Portal SQL Server 7 MS ADO SNITZ Forum Decision Script DiPNET Date Picker IIS Ms IIXSO MS CDONTS SICOD SIANAR Portal Decision Script Questionnaire Oracle Portal

N. of provided Functionality 5 4 5 7 15 9 15 14 28 15 2 7 3 2 18 11 15 4 20

Funct. Cov. Compliance TT NTT 0,222 0,800 14 0,378 0,111 0,500 4 0,108 0,222 0,800 4 0,108 0,222 0,571 0 0,000 0,388 0,466 7 0,189 0,055 0,111 8 0,216 0,200 0,266 5 0,384 0,100 0,142 1 0,076 0,400 0,285 0 0,000 0,050 0,066 0 0,000 0,050 0,500 0 0,000 0,100 0,285 4 0,307 0,050 0,333 2 0,153 0,050 0,500 1 0,076 0,482 0,777 0 0,000 0,344 0,909 0 0,000 0,034 0,066 2 0,117 0,137 1,000 0 0,000 0,379 0,550 15 0,882

Familiarity High Medium High Low Low Medium Medium Medium Low High Low Low Low Low High High Low High Low

Support MME High 30 High 15 High 12 Medium 4 High 20 High 0 High 2 High 1 Medium 1 Low 0 Low 0 High 6 High 2 High 1 High 150 High 60 Low 2 High 20 High 50

NMME 0,370 0,185 0,148 0,049 0,246 0,000 0,153 0,076 0,076 0,000 0,000 0,461 0,153 0,076 0,531 0,212 0,007 0,070 0,177

3.7. Threats 3.5. Collected Data Data were collected in different ways, during project execution and they were stored in the historical database. Project managers provided data concerning Functional coverage, Compliance, TT, Familiarity and MME and the developers provided data concerning support of each component (Component support). Starting from the observed values provided by project managers NTT and NMME have been calculated through the application of the normalization factors. Note that both Familiarity and Support have been scored by the Project Managers according to their own subjective criteria. Table 1 summarizes the collected data concerning components considered in our investigation.

3.6. Research Questions In order to investigate whether the identified characterization measures would have been effective if adopted for components selection for SICOD, DiPNET, and Portal, their values have been analyzed. More precisely, the research questions faced by this study are: - are there some measurable components characteristics, which can support the decision about the choice, with the aim to build a maintainable CBS? - if so, are the relationships due to specific features of the projects, or are they independent from them? As dependent variable we chose NMME, and the independent variables were Functional coverage; Compliance; NTT; Familiarity; Support.

The design of our empirical study allows to obtain results, which are not affected by the following threats: - randomness and irrelevancies of people, tools and processes used within all projects. In fact they have not been chosen by experimenters, but by the project managers with the aim to reach the specific project purposes. So, we can suppose that competences and experiences are randomly present in each project. - heterogeneity of experimental subjects. Each project had a specific history, and developers within this history had a maturation about building and maintenance processes. The influence of history and maturation on the results of this study are present random in all projects. Therefore, the noise produced on the results can be considered without influence. - the treatment is unique for all project. It consists in “using components for building and maintaining CBS”. The obtained CBSs are now running and therefore, we can suppose the treatment as successful. Instead, the following threats could affect the results: - the reliability of measures, because of the subjectivity of Functional Coverage and Compliance, which make use of the concept of “Functionality” and this is not objectively defined; - the set of experimental subjects, even if variegate, is not representative of the population of all software developers and maintainers; - the set of experimental objects, even if variegate, is not representative of the population of all CBSs.

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

3.8. Statistical Analysis

4. Results

Before searching correlations among variables, the mutual independence among the independent variables is verified. If so, we can execute the analysis aimed at: - searching correlations between characterization measures and NMME and - establishing if statistical differences exist among the projects with respect to the characterization measures. In order to establish if a relationship exists between an independent variable v, and a dependent variable w, when both v and w are measured on at least an ordinal scale and normality assumption is not always satisfied, the Spearman analysis can be used. This analysis [24] provides a correlation coefficient ρ, ranging from -1 to +1 and the associated p-level, ranging from 0 to 1. The value of ρ establishes the correlation between v and w: the higher is the absolute value of ρ, the higher is the correlation. The p-level defines the statistical significance of the found correlation: conventionally the p-level values, lower than 0,05 are said statistically significant. If the p-level obtained by the Spearman analysis between v and w is lower than 0.05, then the null hypothesis can be rejected, and therefore a relationship exists between v and w. In our case Spearman analysis is applied for establishing if correlations exist when the dependent variable is NMME and the independent variable is either Support, Familiarity, Functional coverage, Compliance, or NTT. In order to establish if the possible correlations identified depend on specific features of the projects, we tested the statistical differences among the projects. To this end the Kruskal-Wallis Anova by ranks test was adopted. This test provides the statistical significance of the differences, among the values of measures in the three projects. The statistical significance of differences is expressed by the p-level: if it is lower than 0.05, then a statistically significant difference exists among the groups of the independent variable. The Kruskal-Wallis Anova by ranks test can be applied since the following hold [24]: - the normality assumption is not always satisfied; - both dependent and independent samples are random samples from their respective populations; - in addition to independence within each sample, there is mutual independence between the two samples; - the scale for the dependent variable is at least ordinal. In conclusion, thanks to the features of the collected measures: - independence among characterization measures is investigated through the Spearman analysis; - correlations between NMME and characterization measures are investigated by the Spearman analysis; - differences among projects are investigated through Kruskal-Wallis Anova by ranks test.

The tool Statistica 6.0 developed by StatSoft [29] has been used to execute all the analyses.

4.1. Independence measures

among

characterization

Table 2 summarizes the results of the analysis aimed at assessing the mutual independence among the characterization measures. Each cell (i, j), i ≠ j in the table reports the p-level of the correlation between the i-th and the j-th characterization measure, obtained by the Spearman’s analysis; each cell (i, i) is void, in that it does not make sense to establish reflexive correlation. Table 2. Results of the Spearman’s analysis aimed at assessing independence among characterization measures FunCov Compl NTT Familiarity Support 0,056 0,554 0,900 0,238 FunCov 0,056 0,213 0,985 0,272 Compl 0,058 0,186 0, 554 0,213 NTT 0,456 0,985 0,186 Famil. 0,900 0,272 0,058 0,456 Support 0,238 Results show that characterization measures are independent each other, because the values of p-level are always greater than the conventional threshold of 0.05.

4.2. Relationship between characterization measures

NMME

and

Table 3 reports the results of the Spearman’s analysis aimed at establishing whether relationships between NMME and characterization measures exist for each one of the independent variables. It includes the correlation coefficient ρ, the p-level and the last column states if a correlation exists. It results that a correlation exists between each independent variables and NMME, except for Familiarity and Compliance. Table 3. Results of the Spearman’s analysis considering the three projects simultaneously Variables p-level Functional coverage 0,047 Compliance 0,111 NTT 0,008 Familiarity 0,953 Support 0,003

ρ 0,502 0,413 0,634 0,015 -0,687

Correlation YES NO YES NO YES

Note that, for the analysis of NTT, data about the three legacy system included within Portal, i.e., SICOD, SIANAR and Questionnaire, were removed. In fact, the training time spent for those components were 0, but they have been developed as independent projects within the same organization, which later used them as components for Portal. Therefore no training was required.

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

4.3. Differences among projects Standing the results above, the study then established whether significant differences existed among the three projects with respect to the measures correlated to NMME and NMME itself. To this end the following hypotheses have been tested: - H0 null hypothesis: some statistically significant difference exists among SICOD, DiPNET and Portal, with respect to those measures. - H1 alternative hypothesis: no statistically significant difference exists among SICOD, DiPNET and Portal, with respect to those measures. The results of the Kruskal-Wallis Anova test are summarized in Table 4. For each combination of projects (SICOD, DiPNET), (SICOD, Portal), (DiPNET, Portal), and (SICOD, DiPNET, Portal) the table lists the p-level obtained by the test, in order to assess the statistical significance of the differences among the values of each measure, with respect to each combination of projects. Table 4. Results of the Kruskal-Wallis ANOVA p-level value among projects SICODVariable SICODSICODDiPNETFun_Cov NTT Support NMME

DiPNET

Portal

Portal

0,070 0,401 0,4385 0,518

0,738 0,317 0,404 0,504

0,794 0,191 0,695 0,794

DiPNETPortal 0,254 0,321 0,475 0,745

Results show that the Kruskal-Wallis Anova test did not discover any statistically significant difference among those measures. Therefore, we can conclude that the correlations between NMME and Functional Coverage, NTT and Support are independent from specific features of the three projects.

5. Discussion The analysis of data concerning the three industrial projects showed that the time spent for training people is positively correlated to the values of the mean effort spent for maintenance activities. This correlation can be explained considering that the components suppliers propose training periods depending on the intrinsic difficulty in understanding and managing the components: the more difficult is the comprehension of the component the higher maintenance costs it requires. Therefore, we can argue that the proposed training period is a proper characteristic for components selection, in that it provides an indication of the difficulty in understanding, and therefore in maintaining it. A positive correlation exists between Functional Coverage and maintenance effort. So the probability a change in the CBS impacts components is high. This can be explained considering that the higher Functional Coverage for a given component, the higher the

probability that a change in the CBS impacts that component. If the provider does not evolve the component, the impact can have as an effect the need to build new software, aimed to specialize the component for new features of the CBS. This problem can increase when the component provides many functionality. In fact, in such a case, the same change in CBS can have multiple, and different impacts on the same component. Moreover a negative correlation exists between maintenance effort and the level of support provided by the vendor of the component. A possible explanation is that the problems in understanding a given component encountered during maintenance can be faced by the software engineers making use of the provided support. As a consequence, the provided support allows to rapidly solve those problems. The remaining measures do not show relationships with maintenance effort. For what concerns Compliance measure, the explanation can be that, when a component needs to evolve as an effect of the changes of the CBS integrating it, the maintenance effort does not depend on the amount of its usage in the CBS, but it depends on the component itself. A conjecture about this consideration is that this measure can be meaningful during the development of the CBS, when it is worth making use of components with high Compliance. We expected that familiarity had a correlation with respect to NMME. The absence of relationships can be explained noting that this measure does not necessarily imply the knowledge of the components issues involved during maintenance. In other words, high familiarity with a component could be developed on a set of aspects, but when maintaining the CBS further unfamiliar aspects can be required. As a final remark note that the obtained results do not depend on specific feature of one industrial project. The absence of statistically significant differences among the three projects allows to conclude that no one of them is predominant with respect to the others, and therefore, the found features are shared by all of them.

6. Conclusions This paper continues our research aimed at understanding how the component characteristics affect CBSs quality factors. More precisely, in this paper we analyzed whether the identified set of characteristics affects maintainability. Lesson learned 1: CBS functionality provided by each component should be as concentrated as possible over a single aspect of the application domain. If so, the component is impacted only when changes in the CBS concern that aspect. Our hypothesis is that each event requiring maintenance in a CBS probably needs changes in functions included in one aspect. Therefore, each

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

component is impacted when only one aspect is modified. Moreover, if a component covers functionality concerning more “aspects”, its maintenance can be harder because its compliance must be saved in different parts of CBS. Lesson learned 2: The training time offered by the component’s producer usually indicates the complexity of understanding it. If a component is difficult to understand, then it is also difficult to maintain. This explains the relationship between the offered training time and the maintenance effort. Therefore, if the producer acknowledges the need of a high training for a COTS product, then most likely it is difficult to learn and use, therefore its adoption within the target system can be critical. In this case, the costs overhead due to training integration and maintenance should be counterbalanced by a real effectiveness within the system. In conclusion: high training time indicates a complex component, and this implies high maintenance effort. As an effect, when more components provide the same functionality to the CBS, it is worth choosing the component for which lower training is offered. Lesson learned 3: a deep knowledge of the component is necessary for the organization before its adoption, therefore, a trial usage of components is advised for a period of time after the training and before the final decision about their adoption. This avoids the risk that the offered training time is not adequate for comprehending the needed component. In fact, the trial usage can show the training is not sufficient to adequately understand the component. Moreover, this empirical study confirms the importance of the software engineering best practices in providing adequate documentation to all software components. This documentation represents an added value the vendors provide together with the components. As a final remark, further investigations are needed to: - analyze further characteristics and quality factors, which can be meaningful for choosing components - validate the cause – effect relationship between characteristics and maintenance effort. In fact, the current study concerned three heterogeneous projects, which are not statistically representative of all the software projects. These future studies should better take into account the threats that have been poorly considered in this study, in order to remove the suspect that the obtained results are affected by unconsidered factors. More precisely, it will be necessary to organize metrics collection through more effective techniques and tools than interviews. In order to collect data which have not been considered in this study.

7. References [1] Bianchi A., Caivano D., Conradi R., Jaccheri L., Torchiano M., Visaggio G., “COTS Products

Characterization: Proposal and Empirical Assessment”, in R.Conradi and A.I. Wang (Eds.), Empirical Methods and Studies in Software Engineering – Experiences from ESERNET, Lecture Notes in Computer Science 2765 (2003), 233-255. [2] Oberndorf, T., “COTS and Open Systems: An Overview”, CMU-SEI, (2000), url http://www.sei.cmu.edu/str/descriptions/cots.html#110 707 [3] Lawlis, P.K., Mark, K.E., Thomas, D.A., Courtheyn, T., “A Formal Process for Evaluating COTS Software Products”, Computer, May (2001), 58-63. [4] Basili, V., Boehm, B., “COTS Based Systems Top 10 List”, Computer, May (2001), 91-93. [5] Boehm, B., Abts, C., “COTS Integration: Plug and Pray? “, Computer, January (1999) 135-138. [6] Wallnau, K., Brown, A., “A Framework for Evaluating Software Technology”, IEEE Software, 5, (1996) 39-49. [7] Comella-Dorda, S., Dean, J., Morris, E., Oberndorf, T., ”A Process for COTS Software Product Evaluation”, Proc. of 1st International Conference on COTS Based Software Systems (ICCBSS), Orlando (FL), (2002) 86-96. [8] Maiden, N., Ncube, C., ”Acquiring COTS Software Selection Requirements”, IEEE Software, (1998) 4656. [9] Carney, D., Wallnau, K., “A basis for evaluation of commercial software”, Information and Software Technology, 40, (1998) 851-860. [10] Morisio, M., Tsoukiàs, A., “IusWare: A methodology for the evaluation and selection of software products”, IEEE Proceedings-Software, 144, 3, (1997), 162-174. [11] Lawlis, P.K., Mark, K.E., Thomas, D.A., Courtheyn, T., “A Formal Process for Evaluating COTS Software Products”, Computer, May (2001), 58-63. [12] Kitchenham, B., “DESMET: A method for evaluating Software Engineering methods and tools”, Keele University, (1996). [13] Kontio, J., “A Case Study in Applying a Systematic Method for COTS Selection”, proceedings. of the IEEE-ACM 18th International Conference on Software Engineering (ICSE), Berlin, Germany, (1996), 201-209. [14] Ochs, M.A., Pfahl, D., Chrobok-Diening, G., “A Method for Efficient Measurement-based COTS Assessment ad Selection - Method Description and Evaluation Results”, proceedings of the IEEE 7th International Software Metrics Symposium, London, England, (2001), 285-296. [15] Boloix, G., Robillard, P., “A Software System Evaluation Framework”, Computer, 8, (1995) 17-26. [16] CLARiFi Consortium (2002).

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

[17] ISO: Information technology -- Software product evaluation -- Quality characteristics and guidelines for their use. International Organization for Standardization, International Electrotechnical Commission, Geneva (1991) [18] Birk, A., Dingsor, T., Stalhane, T., “Postmortem: Never Leave a Project without It”, IEEE Software, May-June (2002) 43-45. [19] Oracle, available at the url http://www.oracle.com [20] Applix, url http://www.applix.com/index.asp [21] Plumtree: http://www.plumtree.com/default_flash.asp [22] Murphy, J., Higgs, L., Quirk, C., “The Portal Framework: The New Battle for the Enterprise

Desktop”, AMR Research Report, March (2002), available at the url: http://www.amrresearch.com/ [23] Crystal Report, http://www.crystaldecions.com [24] Microsoft, http://www.microsoft.com/ [25] SnitzForum, http://forum.snitz.com/ [26] Decision Script, http://www.vanguardsw.com/ [27] DatePicker, http://www.softricks.com/js/ [28] W.J. Conover, Practical Nonparametric Statistics, John Wiley and Sons, 1980

[29] Statistica available at the url http://www.statsoft.com/

columns list the components integrated in the CBS. If the functionality at the i-th row is provided to the CBS by the component at the j-th column, than the cell (i, j) includes a “X”, otherwise the cell is void. The last row of each table totalize the number of functionality each component offers to the CBS.

Appendix

This Appendix reports on the data discussed in the paper, organized in tables. Tables A1 to A3 summarize the components integrated in SICOD, DiPNET and Portal, respectively. In each table, the first column lists the functionality exploited by the CBS, and the remaining Table A1 Functionality provided by SICOD Data insertion Data modification Data deletion Data browsing Forms visualization for data management in Internet Intercepting user actions on forms for data management in Internet Data browsing for report creation Report visualization Report conversion in pdf format Report printing Databanker for interaction with Oracle DB Interface for interaction with Crystal Report Asynchronous exchange of messages among users Escalation agent, which verifies predefined conditions in the DB and consequently performs predefined actions Visualization of forms for data management in Internet as ASP pages Intercepting user actions on forms for data management in Internet E-mail sending Hosting of the ASP pages produced by Applix Total

Oracle Internet Appl.Server

Oracle DB

Functionality

Oracle Report Crystal Applix Plumtree Builder Report

X X X X X X X X X X

X X X X X X X X X X X

4

The total number of functionality provided by SICOD (Table A1) is 18, and this is the value used for calculating the Functional Coverage of SICOD. Note that both Oracle Report Builder and Crystal Report provide the same set of functionality, that are the functionality of building reports. In fact, both of them have been used to manage reports: the former manages reports concerning Oracle DB data, the latter manages reports concerning Applix data. This choice is the effect of the previous choices of the components managing the data and the CRM, in that Oracle Report Builder is a COTS integrated within the Oracle environment and Crystal Report is the COTS used by Applix to manage reports.

2

4

4

7

X 1

The total number of functionality provided by DiPNET (Table A2) is 20: this is the value used for calculating the Functional Coverage of DiPNET. Each one of those functionality is provided by a different component. The total number of functionality provided by Portal (Table A3) is 29: this is the value used for calculating the Functional Coverage of Portal. Note that also in Portal case, a number of functionality is provided by more than one component. More precisely, data management functionality are provided by both SICOD and SIANAR; and report building functionality are provided by SICOD, SIANAR and Oracle Portal.

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

Table A2 Functionality provided by DiPNET Functionality Data insertion Data modification Data deletion Data browsing Visualization of forms for data management Intercepting user actions on forms for data management Post insertion Forum insertion Post modification Forum modification Post deletion Forum deletion Post browsing Forum browsing Presentation of decisions Presentation of calendar Page publication over the Internet File upload Text retrieval E-mail sending

SQL Server ADO X X X X X X

Total

4

SNITZ Forum Decision Script Date Picker IIS IIXSO CDONTS

X X X X X X X X X X X X X 2

8

1

1

2

1

X 1

Table A3 Functionality provided by Portal Functionality Data insertion Data modification Data deletion Data browsing Single sign-on Web-caching Data browsing for report creation Report visualization Report conversion in pdf format Report printing Form building Graphic chart building Search engine Content management User community management Databanker for interaction with Oracle DB Interface for interaction with Crystal Report Asynchronous exchange of messages among users Escalation agent, which verifies predefined conditions in the DB and consequently performs predefined actions Visualization of the forms for data management in Internet as ASP pages Intercepting of user actions on forms for data management in Internet E-mail sending Hosting of the ASP pages Impact Factor Presentation of decisions Questionnaire design Questionnaire distribution Questionnaire compilation Answers recording Total

SICOD Oracle Portal Decision script Questionnaire SIANAR X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

14

10

Proceedings of the Eighth European Conference on Software Maintenance and Reengineering (CSMR’04) 1534-5351/04 $ 20.00 © 2004 IEEE

1

X X X X 4

11