A Product-Process Dependency Definition Method - CiteSeerX

A Product-Process Dependency Definition Method* Dirk Hamann’, Janne Jarvinen2,Andreas Birk’, Dietmar Pfahl’ 1

2

VTT Electronics P.O. Box 1100 FIN-90571 Oulu Finland Fax: +358 8 551 2320 Emai1: Janne.JarvinenBvtt. fi

Fraunhofer Einrichtung fiir Experimentelles Software Engineering Sauenviesen 6 D-6766 1 Kaiserslautern Germany Fax: +49 6301 707 200 Email: { hamann,birk,pfahl}@iese.fhg.de

and the software has to be developed in very short cycles, taking into account its close relationship with hardware and other product technologies.

Abstract The success of most software companies depends largely on software product quality. High product quality is usually a result of advanced software development processes. Hence, improvement actions should be selected based on sound knowledge about the dependencies between software product quality attributes and software development processes.

For competitive companies, the customer-perceived product quality and the application domain are the driving forces for the improvement of embedded software development. Existing improvement approaches, however, neither are tailored to the specific needs of embedded software development nor focused on product quality requirements. Often, improvement goals are mainly based on achieving conformance to reference standards, which generally fails to produce needed improvements in the organisation products and services, but nevertheless results in high investments. There is a need to establish improvement approaches, which are able to directly impact the organisation’s performance and particularly product quality, and which support an evaluation of investments needed to achieve planned targets.

This article describes a method for developing product/process dependency models (PPDMs) for product-driven software process improvement. The basic idea of the PPDM approach is that there are dependencies between product quality attributes, which are examined according to I S 0 9126, and the software processes, which are assessed with the BOOTSTRAP methodology for example. The Goal-Question-Metric approach is used for product/process dependency hypothesis generation, analysis, and validation. We claim that after jnding and using these dependencies it is possible to focus improvement activities precisely and use resources more efficiently. The approach is currently being applied in three industrial applications in the ESPRITproject PROFES.

*

In the PROFES project these concerns have been addressed resulting in a product focused improvement methodology. As part of the PROFES methodology, a method for developing productiprocess dependency models is presented in this document.

1. Introduction

2. Background

The increasing amount and complexity of software in embedded systems (e.g., telecommunications systems, medical instruments, retailing systems or avionics) sets high requirements for the quality of the products as well as for the management of the development process. The amount of software-related work is often more than 70 percent of the development effort for the whole system,

In the modelling of productiprocess dependencies two ISO-standards are used as background references in the following way.

I S 0 9126 standard [8] for defining the product quality characteristics, and IS0 15504 (also known as SPICE) standard[l2] for defining the basic set of software processes.

Parts of this work were funded by ESPRlT project no. 23239 “PROFES: PROduct Focused improvement of Embedded Software processes”.

898 1089-6503/98$10.00 0 1998 IEEE

The quality characteristics defined in the IS0 9126 standard are applied as such and the IS0 15504 standard is covered by compliant and further developed process definitions of the BOOTSTRAP assessment methodology. The following subsections outline the background models on a generic level.

There exist dependencies between embedded product quality characteristics and certain software (development) processes. It is assumed that after validating the dependencies and applying PPDMs it is possible to focus improvement activities precisely and use resources more efficiently. The IS0 9126 standard is used as a background reference for the product quality dimension, and the I S 0 15504 compliant BOOTSTRAP process model is used for structuring and assessing the process dimension.

2.1 Product quality characteristics and the IS0 9126 standard The I S 0 9126 standard divides software product quality into six characteristics: functionality, reliability, usability, efficiency, maintainability and portability. These characteristics are further decomposed into subcharacteristics and metrics. This breakdown of characteristics into sub-characteristics is not part of the standard but they have been proposed in an appendix, as informative and non-prescriptive guidelines.

The productiprocess relationships are modeled explicitly as Productiprocess Dependency Models (PPDMs). All the process improvement actions proposed by assessments are analysed and a hypothesis of their effect on product quality is given. The actions are complemented and prioritised based on the actual product improvement needs. The effect of process changes to product improvement is evaluated by using the GQM approach [3]. Using the Quality Improvement Paradigm (QIP) i Experience Factory (EF)[2] these validated PPDMs can then be packaged and stored in an experience base and re-used whenever needed.

2.2 Software processes and the BOOTSTRAP methodology The IS0 15504 compliant BOOTSTRAP 3.0 includes a process dimension and a capability dimension. The capability dimension of the BOOTSTRAP 3.0 model consists of six capability levels matching the capability levels of the reference model in part 2 of I S 0 15504 [12]. The BOOTSTRAP 3.0 process dimension is a structured tree that identifies the main categories of process clusters, processes and base practices. The root corresponds to the full architecture. The second level contains the main categories: organisation, methodology and technology. Under organisation and methodology, the process clusters are identified. The process areas are then divided into processes. The base practices belong to each process as basic constituents of the relevant process.

3. Building PPDMs PPDMs are empirical software engineering artifacts that unlike code modules, design documents, defect reports, or measurement data are not readily available as a result or by-product of regular software development activities. PPDMs need to be derived through empirical techniques such as data analysis or knowledge acquisition. During PPDM building there is some degree of freedom on how the resulting PPDMs look like and what information they are supposed to contain. Therefore, sound building of PPDMs requires an understanding of the PPDM lifecycle (i.e., consideration of future usage needs).

2.3 Basic product/process dependency model concepts

3.1 The PPDM lifecycle

In this section, the basic concepts of ProductProcess Dependency Models (PPDMs) are presented. The background assumptions behind product driven process improvement are as follows (cf. Figure 1):

This section outlines the PPDM lifecycle. It provides a baseline for the description of the PPDM building method. Figure 2 depicts the three states of PPDM lifecycle: ( 1 ) PPDM building, (2) PPDM usage, and (3) PPDM evaluation. PPDM building is the creation of PPDMs by uncovering and formalising productiprocess dependencies. The resulting PPDMs are stored in an experience base, which can have an arbitrary form, e.g., a binder, intranet pages, or a database. PPDM usage takes place during planning of improvement programmes. They are supposed to guide the selection of high-potential process changes that are driven by product improvement goals. During the improvement programme and its

Figure 1: Relationship between product and process in PROFES. The better quality the production process has the higher is the quality of the product produced, and

899

software projects, the success of applying the practices and processes that have been selected and implemented based on the PPDMs can be evaluated empirically. The evaluation results can trigger updates of the PPDMs that are contained in the experience base. I

PPDM building in the two phases.

I

Figure 3: PPDM development phases.

3.3 Construction of PPDMs Three different approaches for constructing productiprocess dependencies are described in this section: Interview and survey techniques, data mining techniques, and product and process assessment techniques. Some of these methods including a short description and references for further information are provided for each of the techniques.

Figure 2: The PPDM lifecycle. PPDM building consists of constructing and validating PPDMs. Construction can also be denoted as capturing, because previously hidden and possibly complex causal relationships are made explicit. Basic techniques of PPDM construction are empirical analysis, data mining, and knowledge acquisition. The gained initial PPDMs need to be validated. The two principal validation strategies are empirical analysis (e.g., through case studies) and review (e.g., inspections by experienced software professionals). Reviews can be combined with simulation and testing. PPDM usage consists of characterisation of the reuse context, retrieval of reusable candidate PPDMs, selection of candidate PPDMs, and tailoring. It is part of the planning phase of an improvement programme. The applied practices and processes can then be subject to further investigation. This is done in integration with the improvement programme or software project. These activities represent the evaluation phase. Its results may lead to updates of the experience base and the PPDMs contained in it. In most cases, empirical evaluation needs to be conducted through a case study that can be supported by data from software measurement

3.3.1 Interview and survey techniques. PPDMs are not yet used as an explicit artefact in software engineering. However, the knowledge that is to be contained in PPDMs is implicitly available in the minds of experienced software professionals. It is widely used but not yet described explicitly nor used systematically. For this reason, interview and survey techniques are a very appropriate means to gain PPDM hypotheses. In case that these interviews and surveys utilise state-of-the-art knowledge acquisition approaches, even some first validation of the hypotheses can be done and assured. In addition, in the PROFES project, first PPDM hypotheses have been gained through interviews and surveys. The necessary information has been derived via questionnaires that asked for practices with assumed relevant impact on the product quality characteristic "Reliability" when applied within a certain BOOTSTRAP lifecycle process. The input from the industrial application projects have been processed and given a uniform structure to be comparable. They were presented in a table that lists the respective software engineering practices for each of the BOOTSTRAP processes.

3.2 PPDM building phases This section describes the general approaches for building producb'process dependency models. The PPDM building steps can he assigned to one of the following two

3.3.2 Data mining techniques. This subsection gives a short overview on some well-known data mining techniques as well as an example of one of the techniques. The two "high-level" primary goals of data mining in practice tend to be prediction and description. Prediction involves using some variables or fields in a database to predict unknown or future values of other variables of interest. Description focuses on finding

phases (cf. Figure 3):

Phase 1: Construction of productiprocess dependency models (cf. Section 3.3) and Phase 2: Validation of producVprocess dependency models (cf. Section 3.4). Figure 3 summarises all methods and techniques used for

900

human-interpretable patterns describing the data. The relative importance of prediction and description for particular data mining applications can vary considerably [ 6 ] . There exist a variety of data mining methods. However, it is not intended to provide a complete list of data mining techniques in this document and explain them in detail. For further information in one of‘ the techniques, please refer to the literature. Some of‘ the most powerful data mining techniques are Optimised Set Reduction (OSR) [ 11, Classification Trees[5], and the Rough-Set approach[ 111.

address all of them would exceed the scope of this paper, For this reason, here only the fundamental PPDM validation approach is described rn a somewhat simplified manner. A PPDM expresses a relationship between a process element and a product quality that occurs in real project environments. The PPDMs are empirically valid, if the desired product quality characteristic has been achieved in a software project, and if it can be shown that the product quality achievement was due to the application of the methods, techniques and tools that have been applied in order to fulfil the purpose of the software process in the respective context. The appropriate type of empirical investigation for detailed PPDM evaluation is case study research complemented with measurement data. For the sake of appropriateness and validity of the measurement data it is recommended that it has been gained through GQM-based measurement following the principles described in section 3.4.2.

3.3.3 Product and process assessment techniques. This subsection concentrates on product and process assessment techniques. GQM was the primary technique to assess the product quality in PROFES. The process assessment method used within PROFES is the BOOTSTRAP approach. The BOOTSTRAP approach [9] has already been described in section 2.2. Important for developing hypothetical PPDMs is that the approach introduced a 2dimensional (process vs. capability) framework for process capability ratings. Other important aspects for the development of the improvement recommendations are: The process assessment is expressed as a profile rather than only an aggregated scalar The distance between adjacent levels is divided into quartiles according to the groups of attributes The assessment result is described in a twodimensional profile to increase the granularity of the evaluation.

Empirical analysis of detailed PPDMs requires three sets of empirical data: Baseline data (in the following short: “baseline”), expected observation data (“expectation”), and actual observation data (“observation”). Each data set characterises the four concepts product quality, practice, process, and context of a software project. Baseline data ideally stems from similar, completed project, except that a different (“old”) practice was applied and not the practice recommended by the detailed PPDM. The expectation defines the product quality characteristics that should at least be achieved (by application of the new practice) in order to accept the PPDM hypothesis. The characterisation of process and context should be the same as in the baseline. The actual performance of the applied practice should be characterised by performance indicators that can be matched with the expected performance of the “new” practice as specified in the PPDM. Also, the actually achieved product quality should be expressed through indicators that allow comparison with the baseline. The observation contains the actual data that has been measured when applying the new practice in the current project, The PPDM is accepted as valid, if the observed characteristics of practice, process, and context meet the expectation, and if the expected product quality is met or exceeded.

3.4 Validation of PPDMs The second phase of the PPD building, i.e. PPD validation, can be further refined into the following steps: 1. Perform measurement programme (including collection of data) 2. Compare collected data with hypotheses 3 . Validate PPDMs and explain possible consequences Two approaches for the validation of hypothetical product/process dependency models are described in this section. The first approach is experimental design and the second one uses a GQM analysis framework. The objective of PPDM validation is to support, reject, or modify the initial PPD hypothesis based on empirical data and observations from a software project in which the PPDM has been applied.

It should be noted that the PPDM can also be supported even if the observation does not show the described characteristics. For instance, the PPDM can be supported even if the product quality has not been achieved (namely in the case where other process parameters spoil the positive impact of the practice on product quality; an example is when a good design is spoiled by late changes in the code). This is due to the

3.4.1 Experimental design. Application of a PPDM means that the process element specified in the PPDM has been performed as recommended. Empirical analysis of PPDMs raises several methodological questions. To

901

complexity of real software projects where very many factors interact with each other. It can be managed to some extend by aid of the context characterisation which provides some means to uncover complex interaction effects.

The GQM Abstraction Sheet structure is used: to capture the values (incl. hypotheses about values) of Quality Focus Attributes and Variation Factors after a change in the process (or practice) dimension has been made, and to capture the impact hypothesis. The goal for hypotheses validation could look like the following: e

3.4.2 GQM analysis framework. The GQM approach [3] is used for PPDM building. One technique for knowledge elicitation during measurement planning is the abstraction sheet [ 7 ] . The abstraction sheet is tailored to the specific needs of PPDM building. The results of this tailoring provides a template that serves for: 0 Identifying the elements of candidate producb'process dependencies (i.e., hypothesis generation and baselining of product quality characteristics as well as processes and practices) and Preparing the systematic measurement of product quality characteristics as a prerequisite for quantitative test of product/process dependency hypotheses (i.e., hypotheses validation). The GQM abstraction sheets [4] are used to represent the GQM plans for both phases shown in Figure 3. An example GQM-based template for hypotheses validation is depicted in Figure 4. Quality Focus Variation Factors Total number of defects found in all work products, for each development phase and for each product increment separately. Development phases: (1 ) Specification, (2) Architecture Definition, (3) Analysis, (4) Design,

Analyse the +end product, e.g. mobile phone> For the purpose of

With respect to From the point of view of the *one role, e.g., software

In the context of Gepartment A in the compan,v X Y D

0

There will be about 2000 modification requests in the project over all phases and increments. ...

Relation of new code to old code used Effort spent for different inspections

...

(viewpoint) (environment)

The producb'process dependency hypotheses, which are depicted in the fourth quadrant, are used to describe supposed producb'process dependencies. These hypotheses have the following structure: If the variation factor changes (e.g., increases or decreases), then the quality focus indicator changes (e.g., increases or decreases). The variation factor describes the processes, and the quality focus attribute describes the product characteristics. After having performed the baseline and evaluation measurement activities, it can be decided whether the hypothetical productiprocess dependencies can be accepted or not.

Impact on Baseline Hypotheses The higher the effort spent on inspections, the

less

(quality focus)

The first quadrant (Quality Focus) in Figure 4 contains the set of indicators describing how the examined product quality (reliability in this case) can be measured. The second quadrant (Variation Factors) includes the factors, which are supposed to have an impact on the quality focus. The third quadrant (Baseline Hypotheses) contains hypotheses about possible values and/or distributions for the quality focus attributes defined in the first quadrant. Sometimes, this quadrant contains also real data measured during the baselining activity for the quality focus indicators and hypotheses about expected changes of the quality focus indicators' values after changing something in the development process. The fourth quadrant (Impact on Baseline Hypotheses) is used to set up hypotheses describing the impact of changes of the variation factors on the quality focus attributes (e.g., if variation factor x is lower than during baselining, then product quality indicatory is expected to be higher now).

( 5 ) ...

0

(POW4

engineer>

Defect slippage for each product increment and for each development phase: Additional to the distribution of defects found over severity classes and work product types indicate: Developmentphase in which the defect has been inserted. 0 ...

Baseline Hypotheses (Quality Focus)

(object)

modification

The following product/process dependency hypothesis can be derived from the fourth quadrant (i.e., Impact on Baseline Hypotheses) in the GQM abstraction sheet in Figure 4:

requests will be in later

phases ...

Figure 4: GQM abstraction sheet for PPDM validation.

IH1: If the effort spent on inspection is high, then less errors will be found in later development phases.

902

The hypothesis IHl describes a producVprocess dependency (i.e., between product quality characteristic reliability and BOOTSTRAP process SUP.4 Verification). Thus, product quality is improved by enhancing the inspection technique within the verification process. The hypothesis could have been setup in the first phase of PPDM building (i.e., generate hypothetical product/process dependency models) using one of the techniques introduced in the corresponding section (i.e., section 3.3). In the second phase (i.e., evaluate product/process dependency models), the (dependency) hypothesis has to be validated. This can be done through analysis based on GQM measurement.

The PPDM building process is usually a timeconsuming, multifaceted process that is not only dependent on the organisation that is building the PPDMs, but also on the product and processes in question. There are a number of choices to be made when building PPDMs. The different improvement possibilities must be evaluated and effort concentrated to those activities that will most likely result in efficient results and true improvement.

The effort spent on inspections is defined as a variation factor and the number of defects found in different phases in the development process is defined as a quality focus indicator for product reliability in the GQM abstraction sheet (cf. Figure 4). Each of these factors and indicators is represented by one or more metrics in the GQM plan. During the measurement program the data is collected for all metrics defined in the GQM plan. If enough data is available, it is possible to validate the hypothesis.

It is emphasised that the process of using PPDMs for selection of methods, techniques or tools during planning of software projects or improvement programmes requires the active involvement of experienced personnel both in the managerial and technical level. With the right people, PPDMs can facilitate informed decision making and provide systematic access to explicitly described past experience from other sofiware projects or improvement programmes.

4. Industrial application of PPDM

Acknowledgement

or measurement data) are not readily available as a result or by-product of regular software development activities. PPDMs need to be derived through empirical techniques such as data analysis or knowledge acquisition.

As part of the PROFES methodology, the PPDM

The authors would like to thank all members of the PROFES consortium for many fruitful discussions about the PPDM method. Furthermore, we would like to thank Dr. Martin Verlage and Seija Komi-Sirvib for reviewing an earlier version of the paper.

definition method is being validated and exploited in three parallel industrial case studies representing three different application domains for embedded systems. The industrial application partners Drager Medical Technology (patient monitoring systems), Ericsson LMF (mobile phones), and Schlumberger RPS (retail petroleum systems) have been selected based on a shared set of customer driven product improvement goals, namely product safety, reliability, availability and cost effectiveness. These product improvement goals will be tuned to the individual needs of the application partners. By combining goal-oriented measurement, product and process modeling, reuse of experience and an enhanced embedded systems process assessment approach, the PROFES methodology links software-related product quality factors directly to software development process characteristics and enable continuous product-driven improvement.

References Lionel C. Briand, Victor R. Basili, and Christopher J. Hetmanski. “Developing interpretable models with optimized set reduction for identifying high-risk software components”. IEEE Transactions on Software

Engineering, 19(11):1028-1044, November 1993. Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. “Experience Factory”. In John J. Marciniak, editor, Encyclopaedia of Software Engineering, volume 1, pages 469476. John Wiley & Sons, 1994. Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. “Goal Question Metric Paradigm”. In John J. Marciniak, editor, Encyclopaedia of Software Engineering, volume 1, pages 528-532. John Wiley & Sons, 1994. L. Briand, C. Differding, and D. Rombach. “Practical

5. Conclusions The ProductlProcess Dependency Models (PPDM) reflect past experience about process impact on product quality. This information can be used to guide the identification and design of process parameters for a new, forthcoming project or an improvement programme. PPDMs are empirical software engineering artifacts that (unlike code modules, design documents, defect reports,

guidelines for meusuremmt-based process improvement”.

Software Process Improvement and Practice, 2(4):253280, December, 1996.

Leo Breiman, Jerome H. Friedman, Richard A. Olshen,

903

and Charles J. Stone. “Classification and Regression Trees”. Chapman & Hall, New York, 1984. [6]

Usama M. Fayyad, Gregory Piatetsky-Shapiro, Padhraic Smith, Ramasamy Uthurusamy. “Advances in Knowledge Discovery and Data Mining”. MIT Press, Cambridge, Massachusetts, 1996.

[7]

Christiane Gresse, Barbara Hoisl, and Jiirgen Wust. “A process model for planning GQM-based Measurement”. Technical Report STTI-95-04-E, Software Technology Transfer Initiative, University of Kaiserslautem, 67653 Kaiserslautem, Germany, October 1995.

[8]

ISOiIEC 9126: “Information technology - Sojware product evaluation - Quality chasacteristics and guidelines for their use”. International Organisation for Standardisation (Ed.), Case Postale 56, CH-1211 Geneva,

[9]

Pasi Kuvaja and Adriana Bicego. “BOOTSTRAP - a European assessment methodology”. Software Quality Journal, 3(3):117-127, September 1994.

Switzerland, 1991. First edition 1991-12-15.

[lo] Frank van Latum, Rini van Solingen, Markku Oivo, Barbara Hoisl, Dieter Rombach, and Giinther Ruhe. “Adopting GQM-Based Measurement in an Industrial Environment”. IEEE Software, 15(1):78-86, January

1998. [ 1I] Zdzislaw

Pawlak, Jerzy Grzymala-Busse, Roman Slowinski, and Wojciech Ziarko. “Rough sets”. Communications of the ACM, 38(11):89-95, November 1995.

[I21 ISO/IEC: “ S o f l a r e Process Assessment - Part 2: A Reference Model ,fos Processes and Process Capability”.

Working Draft (Revised), Version 2.0, International Organisation for Standardisation (Ed.), Case Postale 56, CH-1211 Geneva, Switzerland, July 1996.

904