Establishing Product Process Dependencies in SPI - CiteSeerX

Establishing Product Process Dependencies in SPI 1, 2

Markku Oivo

2

1

3

4

, Andreas Birk , Seija Komi-Sirviö , Pasi Kuvaja , Rini van Solingen

1

VTT Electronics, P.O. Box 1100, FIN-90571 Oulu, Finland, {Markku.Oivo, Seija.Komi-Sirvio}@vtt.fi

2

Fraunhofer Einrichtung Experimentelles Software Engineering, Sauerwiesen 6, Technopark II, D-6766, Kaiserslautern, Germany, [email protected]

3

Department of Information Processing Science, University of Oulu, Oulu, Finland, [email protected]

4

Tokheim, Industrieweg 5, NL-5531Bladel, The Netherlands, [email protected]

This paper has been published in the Proceedings of the Fourth annual European SEPG conference, June 7-10 1999, Amsterdam, The Netherlands

Abstract One of the basic assumptions behind most software process improvement methods is that improved processes lead to improved product quality. Despite the popularity of SPI methods, there is a lack of both methods and experience reports on the explicit relationship between software process improvement (SPI) and product improvement. PROFES is a methodology that helps to shift from generic process improvement towards focused improvement of the software processes based on explicit product quality requirements. It combines and enhances methods such as goal-oriented measurement (GQM), process assessment (ISO15504), product and process modelling, and experience factory. The integration of these methodologies helps to focus improvement activities and to use resources more efficiently. PROFES introduces a new method for establishing product process dependencies (PPD), which are used to describe the relationship and interdependency between process and product quality. PPDs are a core element of the PROFES method, and are to date unique in the world of SPI. The initial PPD models were constructed in three industrial organisations, which offered real-life experimental environments for methodology development and validation. The PROFES methodology is customisable, so that it can easily be adopted by different companies for different software development environments, while taking advantage of the investments already made in process improvement, e.g. an established CMM assessment culture. It enables an organisation to focus improvement actions precisely on those parts and characteristics of the process that are critical for product quality. It is supported by a book, user manual, operational guidelines, templates and tools in order to plan and implement product quality-oriented SPI.

1. Introduction Software process assessment, process and product measurement, and process modelling are all important tools for software process improvement (SPI) professionals. On the other hand, there has also been discussion on their relative merits and interrelationships. Furthermore, the use of different methods has been fairly isolated, even in progressive organisations that use several methods in their SPI activities. Methods have often been seen as alternatives and even competitors, although this attitude is not as common as it used to be. The basic assumption in most SPI methods is that improved processes result in improved products. Most SPI professionals agree on this hypothesis. However, there have been some doubts concerning the unquestioned effectiveness of process improvement activities, with regard to improving product quality and even the process itself. People have raised several questions that still remain open issues: • Is it really cost-effective to improve process maturity levels step-by-step, and does such improvement automatically lead to better product quality? • Are measurements really cost-effective, and how should measurements be made? • In order to be able to really benefit from measurements, what should the process maturity level be? • What is the role of process modelling in process improvement? There is no doubt that the business objective of any company is to sell quality products and produce them efficiently. Process improvement is simply a means to this end, not an end in itself. Despite this self-evident fact, there is little hard evidence in the literature about the effectiveness of process improvement methods in final product quality. There is a clear need for a method that can help to focus the process improvement actions precisely on those process attributes that directly affect product quality. No existing methods explicitly supports this fundamental requirement. The objective of the PROFES methodology is to support product focused process improvement. This method changes the traditional unidirectional relationship

Process improvement ⇒ Product improvement into a bi-directional relationship

Process improvement ⇔ Product improvement

PROFES provides help in answering question like "if I want to improve these aspects of my product quality, what should I improve in my process?". More specifically, PROFES supports SPI practitioners in answering such questions as: • Where should I invest, and what should I do to improve the quality of my product? • How can I improve product quality control, delivery time, and production costs? • How can I monitor and control improvement action? • How can I learn from experience? • How much is it going to cost? This paper includes an introduction to the PROFES methodology with industrial experience from real-world projects by three industrial companies over a period of more than two years. The results of using the PROFES improvement methodology have already led to considerable benefits for these companies.

2. Background Elements The PROFES improvement methodology combines, enhances and integrates software process assessment, software measurements, product and process dependency relationships, and experience factory. Software process assessment is based on the ISO 15504 for software process assessment1 [3]. Software measurements are goal-oriented and based on the goal-question-metrics (GQM) approach [2]. The ISO 9126 [7] is used as the background reference for the product quality characteristics when defining the product-process relationships. The integration of the methods is aligned with the Quality Improvement Paradigm (QIP) [16] that forms the overall structure of the methodology. The concept of experience factory [16] is applied in defining the procedures for packaging the results of the improvement efforts. 2.1

The BOOTSTRAP assessment methodology

BOOTSTRAP [5] software process assessment methodology was used in the PROFES methodology as it is a full-scale assessment methodology that is fully conformant to the requirements stated by the ISO 15504. Assessment is used to characterise the target software producing unit and its projects, and to evaluate their capabilities for finding process improvement areas and to define improvement recommendations. The BOOTSTRAP methodology, includes reference model against which the process capability is evaluated, assessment method, that defines the assessment procedure, and improvement method that includes guidance how the assessment results are used for process improvement. The BOOTSTRAP reference model includes two dimensions: process dimension and capability dimension. In the assessment the process dimension is used for identifying the objects of evaluation and the capability dimension provides the evaluation criterium. The process dimension of the BOOTSTRAP methodology includes processes that fulfil the requirements of the ISO standards numbers 9001, 9000-3, 12207 and 15504, the European Space Agency standard number PSS-005 and CMM [11]. The capability dimension of the BOOTSTRAP reference model is aligned with the capability concepts of the ISO standard number 15504, including six capability levels. All capability levels apply to each process of the process dimension thus allowing a common improvement strategy. In addition to the ISO 15504 conformant output profiles, the BOOTSTRAP method generates also synthetic profiles using quartiles within the capability levels. 2.2

The Goal-Question-Metrics approach

Goal-oriented measurement [2] represents a systematic approach for tailoring and integrating objectives of an organisation into measurement goals and their stepwise refinement into measurable values. The GQM method was chosen as an element of the PROFES improvement methodology as it is the most mature and widely used measurement approach in place today [12], [2]. Another reason was that GQM approach starts with goal definition and align subsequent improvement actions to the goals.

The GQM method is divided into several stages. After the pre-study, the next stage is to identify a set of measurable quality goals. After the goals have been set, questions that define the goals are derived as completely as possible. The next step consists of specifying the measures, which need to be collected in order to answer the questions defined, and to track the conformance of products and processes to the defined measurable quality goals. The necessary information is collected using structural interviews with 1

Also sometimes called as “SPICE” standard according to the International project called SPICE (Software Process Improvement and Capability dEtermination).

experienced project people. The GQM interviews use a specific abstraction sheet to structure the obtained information during the interviews. Defined goals, questions and metrics are described in the GQM plan. After the measurements have been specified, a mechanism needed to collect measurement data is developed. This is described in the measurement plan and in the associated data collection forms. The data is then collected and validated during the software development project according to the measurement plan. The collected data is analysed and discussed in feedback sessions [12]. Feedback sessions are organised meetings involving members of the project team and the measurement team. Project team consists of the software engineers of the target project that is under improvement. Feedback sessions are an essential mechanism supporting analysis and interpretation of the measurement results. After the end of the software development project all relevant information gathered during the project has to be packaged and stored for later retrieval and reuse. This is very important for continuous learning and improvement. 2.3

The software product characteristics

One of the main targets of the PROFES project is to develop a product quality improvement methodology that is driven by the product quality characteristics. In order to achieve the goal the product quality characteristics are connected to process characteristics through product-process dependency relationships. The ISO standard number 9126 [7] was chosen in the PROFES project for the starting point to define the quality characteristics of the embedded software product. The standard divides software product quality into six characteristics that are functionality, reliability, usability, efficiency, maintainability and portability. 2.4

Quality Improvement Paradigm

A general Quality Improvement Paradigm (QIP) [16] for software quality was chosen as one of the background approaches to figure the overall structure of the PROFES improvement methodology. QIP is a heuristics that applies iterative procedure for software quality improvement through systematic use of feedback information that is collected using goal-oriented measurements. The improvement process according to the QIP heuristics is organised into organisational and project level cycles. The process starts on organisational level and proceeds into project level. The project level cycle (control cycle) provides direct feedback to the project during the execution phase. The feedback is used for monitoring the project and indicating changes and improvements in the project performance. The organisational level cycle (learning cycle) provides feedback to the organisation for two purposes. The first purpose is the get analytical information about project performance at project completion for comparing the project data with the nominal range in the organisation and analysing concordance and discrepancy. The second purpose is to collect reusable improvement experiences that are applicable in other projects. The learning cycle contains the following steps: characterise, set goals, choose process, execute and measure, analyse, and package. 2.5

Experience factory

The Experience Factory is an infrastructure for organisational learning about software development. Its main part is the experience base, that is a corporate repository for storing relevant software engineering experiences. It includes project organisation in which the software development activities are performed, and organisational improvement infrastructure, where analysis and packaging activities are carried out for project support and for maintenance and continuous evolution of the experience base. The experience factory supports reuse of good software engineering practices within an organisation once such practices have proven to be effective. Therefore, it can be effectively applied in combination with measurement programmes. As it is an organisational infrastructure it fits ideally for collecting and making available any type of experiences.

3. The PROFES Methodology 3.1.

Overview

Until recently, one of the basic assumptions behind most software process improvement methods is that improved processes lead to improved product quality. Countless numbers of improvement programmes have been executed to improve software process quality all over the world. However, there has been no systematic investigation into the precise effects of specific process improvement action on customer-oriented product quality requirements. Therefore, there is an urgent need to understand and utilise the existing relationship between process improvement and product quality. PROFES (PROduct Focused improvement of Embedded Software processes) [1] is a methodology that helps to shift from generic process improvement towards focused improvement of the software processes based on explicit product quality needs. PROFES offers a solid framework of selected methods, tools, and templates to achieve the product quality required. The PROFES improvement methodology:

• Combines and enhances the strengths of goal-oriented measurement, process assessment, product and process modelling, and experience factory, • Links customer driven product quality needs directly to the process characteristics • Enables an organisation to focus improvement actions precisely on those parts and characteristics of the process that are critical for achieving required product quality, and • Provides support for PROFES usage in a form of the PROFES book, User Manual, tools, training, and presentation material. The PROFES improvement methodology combines and enhances the strengths of widely used methods such as goaloriented measurement, process assessment, product and process modelling, and experience factory. The goal-oriented measurement methodology GQM (Goal/Question/Metric) and the ISO 15504 compliant process assessment and improvement methodology (BOOTSTRAP during PROFES project) provide a framework for the software processes. The ISO 9126 standard is used as a background reference for the product quality characteristics, but the actual product measurements are based on the GQM method. Process modelling is required for description of the software development processes. All these elements are usually applied separately, or are even seen as alternatives. Integration of these methods helps to focus improvement activities and allows a more efficient use of resources. This means for example combining assessment and measurement so that less time and effort is needed than when implementing these activities separately [8]. Data retrieved from measurement programmes is used for both monitoring improvements and also for process assessment. Existing process improvement approaches have failed to use the interdependency between processes and product quality for focusing improvement initiatives on those parts of the processes that are critical for achieving the product quality required. Therefore, one of the main goals of PROFES was to formulate a method for finding and describing the up to now unused opportunity for efficient process improvement: the explicit link between the product quality required, and the processes that influence it. Therefore, PROFES introduces a new method for establishing product process dependencies (PPDs) [9] [10], which are used to describe the relationships and dependencies between process and product quality. PPDs are a core element of the PROFES method, and are to date unique in the world of software process improvement (SPI). The initial PPD models were constructed in three industrial organisations, which offered real-life experimental environments for methodology development and validation. Development of the PROFES improvement methodology is carried out in the domain of embedded software, i.e. in software-producing organisations working in the field of embedded systems. Embedded systems are computers incorporated in products and electronic systems that implement intelligent functions such as control and communication. Examples of products relying on embedded systems are mobile phones, medical instruments, and fuel dispensers. Embedded systems are the main application area for PROFES, but elements are also applicable in other fields of software development. The PROFES methodology is a customisable environment, and can take advantage of the investments already made in process improvement, for example in an established CMM assessment culture. It enables an organisation to focus improvement actions precisely on those parts and characteristics of the process that are critical for product quality. Beside methodology development, PROFES has built and enhanced tools and templates to support practical product quality-oriented, process improvement work. Here PROFES has enhanced and linked selected tools together to support definition of goal-oriented metrics; retrieving, managing and analysing collected measurements according to a measurement plan; and analysing improvement actions trends. Document templates are enhanced, partly based on methods that PROFES utilises and partly generated only for PROFES specific purposes, e.g. templates for product process dependency definition and presentation. The PROFES improvement methodology is supported by operational guidelines, templates, and tools to plan and implement product quality-oriented SPI. The components of the PROFES methodology kit include a User Manual, a PROFES book, tools, training material and presentation material. Previously, improvement has been focused on improving the processes, for example to reach a higher maturity level, while the link to product quality has been obscure or totally ignored. When process improvement initiatives are explicitly linked to product characteristics, the effects of improvement actions are more visible in the form of increased product quality. Using PROFES improvement methodology provides the answers to the difficult questions that managers meet in their every day work: • Where to invest? • How much is it going to cost? • How to monitor and control improvements? • How to learn from experiences? The PROFES improvement cycle and PPDs are elaborated in the following sections and other elements are briefly described and references to further information are given.

3.2.

PROFES improvement cycle

The PROFES improvement methodology uses a modified version of the Quality Improvement Paradigm in order to meet the goals for product driven process improvement. To illustrate and emphasise the importance of the product as a driver for improvement, it is placed in the middle of the PROFES improvement circle (see Figure 1). The product is the starting point for any improvement activity, starting with the identification of the product quality needs and the determination of the preliminary product quality goals. Product-Process Dependencies (PPD) form the linking element between the product and the product development processes. PPDs make the relations between parts of the software development process and product quality explicit. PPD models are used to find and determine the required process changes to achieve stated product quality improvement goals. Ch ara cte ris e

Set goals

Analyse

ge cka Pa

Product

PPD

Ex ec ute

Process n Pla

Figure 1: PROFES improvement cycle The PROFES method consists of six main phases. These phases are described shortly below. Characterise Product driven process improvement begins with an identification or review of product quality needs, which are gathered from customer surveys, market research, or from other sources. Based on the product quality needs, preliminary product quality goals are set and aligned with high-level goals, e.g. business goals. Characterising includes organisational and project-level assessment and modelling to find out what is the current status of the processes. Potential process changes are identified through assessment and modelling activities. In addition to the process assessment, a product assessment can provide key information in such a product driven process improvement approach. If product related measurement is feasible, product quality characterisation can be done using Goal/Question/Metric (GQM) -based measurement, or some other available method. The results of process and product characterisation are the starting point for setting the improvement goal. Set goals After the assessments, the improvement goals are set based on the current status of both process and product, and the product quality needs that have to be met. Based on the product quality improvement goals a set of potential ProductProcess Dependency (PPD) experience packages are identified. These experience packages can be retrieved from an experience base that provides the PPD models. If there is no such experience base, an analysis of project history may provide preliminary PPD model potential for experience packaging. By modelling and applying these PPD models, the experience base can grow through each consecutive project cycle. The PPD experience packages are verified PPD models. The PPD experience packages contain suggestions of process improvements that are further studied from the viewpoint of the process assessment results, and a selection is made based on criteria such as feasibility. Product improvement goal(s) are transformed into measurement goals. The measurement goals are refined into measures via questions according to the GQM method. The main purpose of the defined measures is to follow and continuously analyse the realisation of product improvement and process change, and to draw final conclusions. Plan Improvement is planned properly before project execution. In this phase needed process changes are selected, described and modelled. A measurement plan is determined, based on measures defined in the GQM plan. The measurement plan includes specifics about the measurement process, measurement frequency, information sources, and all additional information to completely measure the goals defined in the GQM plan. Execute The defined improvement actions are implemented in the development process. This means implementing a process with prescriptive process models that aim to achieve the desired product quality. Measurement activities are established, based on the GQM and measurement plan. Measurements are collected and product quality is continuously analysed in GQM feedback sessions during project execution. Measurement actions may include use of a measurement system that enables continuous measurement and facilitates on-line assessment.

Analyse The purpose of the analysis phase is to analyse whether product quality has improved as predicted, according to changes made to the process. Results are analysed, based on measurements conducted in the execute phase, and corrective actions may be determined. In the analysis phase, the released product data and process data and findings are analysed and interpreted as thoroughly as possible, leading to an update of the relevant models. The differences between product and process-related plans and realisation are analysed, and root causes of deviations are identified. Also the new process models adopted for use are evaluated. This may require a re-assessment of the altered processes. This phase emphasises the gathering of lessons learned during improvement actions. Organisation-wide learning is assured by the organised reuse of this new information. Package The purpose of the package phase is to store all experience gained in the project regarding product-process interdependency. This also includes rejection and modifications of PPD experience packages, which have been concluded in the analysis phase. The storage of experience is necessary for its later reuse in forthcoming projects. Project evaluation findings are documented in a reusable format. Project and organisation-specific terms are removed, and the contextual situation in which the PPD experience packages are supposed to be reused, is defined. The experience gained in the development project is stored in the Experience Base for each PPD experience package. 3.3.

Product Process Dependencies

Repositories of product/process dependencies (PPDs) are a core element in PROFES improvement methodology. They contain an organised collection of so-called PPD models, which are a new software engineering artefact introduced by PROFES. PPDs describe the impact that software engineering processes and technologies have on software quality. They assist the identification of improvement actions, and so make the product-focused process improvement approach of PROFES unique and particularly effective. PROFES has built a large collection of PPDs and made them available to the public in the form of a web-based PPD repository. The repository can be queried over the Internet. It can also be used as the seed for a company-specific PPD repository. Therefore, it can be customised and integrated to suit the needs of individual software organisations or projects. This section introduces PPD models and outlines on how they can be obtained, used, and evolved. PPD Models PPD models describe the impact that software processes – and software engineering technologies that are applied within these processes – have on software product quality. A PPD model example is show in Figure 2. It documents the impact of software inspections (i.e. a software engineering technology) on reliability (i.e., a product quality) when applied within software requirements analysis (i.e., a process). An important part of a PPD model is the context model. It describes in which contextual situations – or, in general, under which constraints – the stated impact on product quality can be expected. In the example, the described impact of software inspections is linked to mid-sized inspection teams with high experience, small or average document size, high management commitment, etc. PPD Model Product Quality Process Practice CF.1 CF.2 CF.3 CF.4 CF.5 CF.6 CF.7 CF.8 CF.9

Reliability ENG.3 Software Requirements Analysis Software Inspections Context Size of inspection team 1-2 3-5 6-8 9-10 Experience of inspection team low average high Problem treatment of inspection team pragmatic detailed Complexity of inspected document low average high very_high Size of inspected document small average large very_large Management commitment low high Overall time pressure low average high Module affected by new hardware old_hw new_hw Module developed externally internally externally

Figure 2: Example PPD model. Developing PPD Models Developing PPD models is a combined analysis and design task. The objective is to identify and analyse relevant software engineering phenomena, and to document them in a way that is most suitable for future knowledge reuse. The information contained in PPD models can be obtained using different strategies and information sources. The most relevant information sources are the following: Interviews with experienced software professionals, software measurement programmes, other project analysis methods, and project documentation, literature and surveys. Interviews provide the most comprehensive and deepest insight into company-specific product/process dependencies. However, their effectiveness depends quite largely on the preparation and background knowledge of the interviewer. For this reason, software measurement programmes and other project analysis methods or project documentation, are important sources of first PPD information (e.g. process assessments, project post-mortem meetings, project files, etc.). For identifying generic, i.e. not company-specific PPDs, literature and surveys can provide good baseline information.

Measurement data is also important for empirical validation of existing PPD models. Its use during PPD development has been described in [13]. It is recommended to combine multiple strategies for PPD development, depending on what information is available. For instance, comprehensive information from literature may already exist, or even PPD models from literature might already be available. These—usually generic models—can then be complemented and refined, based on past measurement data and results from a process assessment. For final consolidation of the new PPDs, they should be reviewed through interview with experienced software engineers. The development of PPDs should be combined with the development of a PPD repository. A PPD repository involves the collection of technology definitions and appropriate taxonomies or index structures (e.g. taxonomies of processes and product qualities). Appropriate taxonomies facilitate the retrieval of PPDs. PROFES has provided a baseline PPD repository. It can be reused by software organisations and customised to meet their specific needs. Using PPD Models The three main usage purposes of PPD models in product-focused process improvement are: • Identification of improvement actions. For a given product quality goal, PPD models help to identify those potential improvement actions (i.e. software engineering technologies to be applied) that are most promising for achieving the required product quality goal. • Process assessments focusing. Process assessments can be quite time-consuming. PPD models help to focus process assessments on those processes that are most critical for a set of relevant product qualities. • Organising repositories of good software engineering practice. PPD models link software engineering technologies or practices with those processes and product qualities to which they can be applied. They also provide information on the contextual situations in which the technologies can be reused most effectively. This information is useful for organising repositories of reusable software engineering technologies. It ensures effective knowledge retrieval from these repositories, and supports informed decision making during the planning and execution of software projects and improvement programmes. In addition, PPD models can be used for supporting software engineering technology transfer, and for the management of project risks that are associated with the application of software engineering technologies. Within product-focused process improvement, the most relevant usage of PPDs is the identification of potential improvement actions and the focusing of process assessments based on previously determined product quality goals. In the following, the use of PPD models for selecting improvement actions is outlined briefly. It requires that a PPD repository is available. For instance, the PROFES PPD repository can be used as a starting point. The identification of potential improvement actions is usually conducted during the planning of a new software project, onto which new product quality requirements are imposed, and which can not be fulfilled with the established processes. A PPD repository can suggest potential technologies for use in the forthcoming project, which are expected to help yielding the set of product quality goals. It also helps in selecting those technologies that best suit the context of the forthcoming software project. Therefore, the PPD's context models assure that the most effective technologies are selected. Using PPDs, the selection of improvement actions for a forthcoming software project can proceed as follows: 1. 2.

Identify product quality goal. Identify processes for which improvement actions are most required.

a. Use the PPD repository for identifying those processes that have particular impact on the selected product quality goal. 3. 4. 5. 6. 7.

b. Use process assessments for identifying those processes of the software organisation that have the highest improvement potential. Retrieve those PPD models from the PPD repository that address the previously determined combination of product quality goal and processes. Construct a characterisation questionnaire from the context models of the retrieved PPD models. Characterise the forthcoming software project for which improvement action is required using the characterisation questionnaire. Rank the previously retrieved PPD models with regard to their similarity to the project characteristics. Select those technologies for introduction to the forthcoming project (i.e., as improvement actions) whose PPD models are most similar to the project characteristics.

Using this strategy, PPD models allow for decision support and informed decision-making during the selection of improvement actions. The decision support process might also deploy the review of records from past projects in which the potential technologies were applied. Such records can be stored as part of a PPD repository. Additional information about technology selection processes and possible tool support is described in the PROFES user manual [15]. Prototypical tool support has been introduced by Birk and Kröschel [14].

Evolving PPD Models As the context of software development evolves, and new experience about the relationships between product quality and the software processes is gained, the PPD models need to be updated and refined in order to represent the evolved context and experience. For this reason, it is recommended that the application of software engineering technologies that have been selected as improvement actions based on PPDs, be monitored. This monitoring is to provide information on whether the improvement actions have been successful, i.e. whether the required product quality has been achieved, and whether the PPD models' application context definition is consistent with the actual application context of the technology in the respective software project. An appropriate technique for conducting this PPD monitoring are software measurement programmes [13].

4. Industrial results This PROFES improvement methodology has been applied in three organisations developing embedded systems: Dräger Medical Technology, Ericsson, and Tokheim, each representing a different industry, i.e. telecommunication, medical systems, and petroleum retailing. The main message from these applications is that PROFES really helps in focusing to the product areas that have improvement priority. The companies all strongly agree that effort is spent only on those product attributes that are relevant. Quality of the final product is the central objective, which is highly appreciated since the embedded system itself is being sold, and not the development process that created it. The PROFES methodology is intended to be customisable to suit an organisation or project. This is necessary because each company or project has its own objectives to be reached. A ‘one-size-fits-all’ method to product focused SPI is not expected to fulfil these individual needs. For example, it is quite common for a department to have an objective such as ‘reaching level 2 at the end of next year’, or a totally different objective such as ‘the first version of the product should have less than five critical defects in the field test’. These two types of objectives are so different that an improvement approach should also support them individually. The PROFES methodology contains several components such as assessments, measurement, experience factory, process modelling, etc. that are available as a type of toolbox. From this set of components an organisation can select the mix that suits it the best. For example, both Ericsson and Tokheim had already carried out several process assessments before PROFES was adopted, thus creating a change in their improvement programme, instead of starting it from scratch. Another example is that both Dräger and Tokheim already had experience with GQM measurement programmes that made the introduction of GQM easier. Each organisation's starting point, together with the (product) objective, require different methods for implementing an improvement programme. The PROFES methodology supports these differences very well. The overall improvement cycle in Figure 1 only describes the general idea of improvement. The individual components are not described. On the other hand, the PROFES methodology is sufficiently flexible for practical application, which was experienced in practice with PROFES at Dräger, Ericsson and Tokheim. The general findings from PROFES application at the three companies are listed hereafter: PROFES methodology is highly usable. During application of the PROFES methodology it appeared to be an approach that is highly usable in practice. None of the companies had problems in applying the methodology. Customisability of the methodology is a major benefit. A strong point of PROFES is that it is a customisable approach to process improvement. The methodology can be applied according to company objectives. Some companies may have a strong focus for improvement along a scale such as CMM or BOOTSTRAP. This is very well supported by PROFES. On the other hand, there are companies that use a product-centred approach, which is also very well supported by PROFES. Product-focused improvements are feasible and do pay off. Centring all process changes around the expected impact on the product is feasible. This is a critical success factor, especially in the embedded systems area. The link between process and product is constantly being evaluated and analysed. There is still a lot to learn about product process dependencies (PPDs). The way in which PPDs work in practice, how their effects can be optimised, and which critical context factors are present, is still unknown. The effects of PPDs appear to differ between organisations: what works for one organisation does not necessarily work for another. Past project experience provides sound input for PPDs. Organisation and projects motivated in applying PROFES elements. The project teams supported the application of PROFES. This was due to the methodology components, to the goal-orientated character, and to product orientation. Recommendations from assessments were useful. The proposed improvement changes from the assessments were clearly accepted, not only for the individual projects but also at the organisational level. Feedback sessions were valuable. Measurements provoke discussions within project teams, and facilitate the group learning process. However, more time is needed for measurement data analysis. The full integration of measurement in the PROFES methodology made available data already useful during the project.

Due to PROFES, the embedded systems industry can now apply a product focus to their process improvement programmes. Embedded system producers sell products, not processes. These products consist not only of software but also of hardware. Improvements in the process should therefore always be aimed at an improvement in the product area: cost, quality or timeliness. The PROFES methodology helps in focusing those improvement areas that are relevant for the specific organisation or project. Effort is spent only on product attributes that have priority for improvement. The PROFES methodology proved to be a useful and effective approach to apply product-focused SPI in industrial companies. Based on the specific needs of the individual company and the specific development project, the applied improvement approach was customised to be fully supported by the PROFES methodology. This is considered to be the greatest benefit of the PROFES methodology. Results in the companies were excellent. Dräger was able to develop their product exactly according to a very tight schedule, and this product was very positively received by hospital staff during clinical tests. In less than one year, Dräger also increased its development project maturity almost from level 2 to level 3. Ericsson delivered their product with a design quality higher than their baseline. Tokheim supported a reliability critical project with a product reliabilityfocused process improvement programme, resulting in just two defects.

Conclusions There is an urgent need to understand how customer-defined product quality characteristics and improvement needs can be linked to specific process quality characteristics and improvements. PROFES is a methodology that helps to shift from generic process improvement towards focused improvement of the software processes based on explicit product quality needs. The main application area for PROFES are embedded systems, but most elements are also applicable in other fields of software development. PROFES combines and enhances the strengths of widely used methods like goal-oriented measurement, process assessment, product and process modelling, and experience factory. Goal-oriented measurement methodology GQM and process assessment and improvement methodology provides the framework for the software processes. The ISO 9126 standard is used as a background reference for the product quality characteristics, but the actual product measurements are based on the GQM method. Process modelling is required for the description of software development processes. All these elements are usually applied separately, or are even seen as alternatives. The integration of these methodologies helps to focus improvement activities and to use resources more efficiently. The PROFES methodology introduces a new method for establishing product process dependencies (PPD), which are used to describe the relationships and dependencies between process and product quality. PPDs are a core element of the PROFES method and are to date unique in the world of software process improvement (SPI). The initial PPD models were constructed in three industrial organisations, which offered real-life experimental environments for methodology development and validation. PROFES can be easily adopted for use in different organisations and different software development environments, for example in an organisation that has used CMM for a long time. PROFES enables an organisation to focus improvement actions precisely on those parts and characteristics of the process that are critical for product quality. It is supported by operational guidelines, templates, and tools to plan and implement product quality-orientated SPI. The components of the PROFES methodology kit include a User Manual, a PROFES book, tools, training material, and presentation material. PROFES is a result of a major European SPI project carried out by highly skilled methodology providers and practitioners with comprehensive expertise in process improvement: Dräger Medical Electronics, Ericsson, Etnoteam S.p.A., Fraunhofer IESE, Tokheim RPS, University of Oulu, and VTT Electronics. Use of the PROFES improvement methodology have already led to considerable improvements: a reduced number of failures in the field tests, product delivery on time with positive customer response, improved design quality, and improvement of the BOOTSTRAP maturity level from level 1 to level 3 in only one year.

References [1] [2] [3]

Andreas Birk, Janne Järvinen, Seija Komi-Sirviö, Pasi Kuvaja, Markku Oivo, Dietmar Pfahl. PROFES – A Product-driven Process Improvement Methodology, In Proceedings of the Fourth European Software Process Improvement Conference (SPI ‘98), Monte Carlo, Monaco, December 1998. Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. Goal Question Metric Paradigm. In John J. Marciniak, editor, Encyclopaedia of Software Engineering, Vol. 1, pages 528–532. John Wiley & Sons, 1994. ISO/IEC TR 15504-2: Information Technology – Software Process Assessment – Part 2: A Reference Model for Processes and Process Capability”. Technical Report type 2, International Organisation for Standardisation (Ed.), Case Postale 56, CH-1211 Geneva, Switzerland, 1998.

[4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

[14] [15] [16]

Khaled El Emam, Jean-Normand Drouin, Walcélio Melo. SPICE The Theory and Practice of Software Process Improvement and Capability Determination. IEEE Computer Society Press, 1998 Pasi Kuvaja, Jouni Similä, Lech Krzanik, Adriana Bicego, Samuli Saukkonen and Günter. Koch. Software Process Assessment & Improvement – The BOOTSTRAP Approach. Blackwell Publishers, 1994. Adriana Bicego, Munish Khurana, and Pasi Kuvaja. BOOTSTRAP 3.0 – Software Process Assessment Methodology. In Proceedings of the SQM ’98, 1998. ISO/IEC 9126: “Information technology – Software product evaluation – Quality characteristics and guidelines for their use”. International Organisation for Standardisation (Ed.), Case Postale 56, CH-1211 Geneva, Switzerland, 1991. First edition 1991–12–15. Matias Vierimaa, Dirk Hamann, Seija Komi-Sirviö, Andreas Birk, Janne Järvinen, Pasi Kuvaja. Integrated Use of Software Assessments and Measurements. To be published in Proceedings of the SEKE'99, 1999. Dirk Hamann, Janne Järvinen, Andreas Birk, and Dietmar Pfahl. A Product-Process Dependency Definition Method. In Proceedings of the 24th EUROMICRO Conference: Workshop on Software Process and Product Improvement, Vol. II, pages 898-904, IEEE Computer Society Press, 1998. Dirk Hamann, Dietmar Pfahl, Janne Järvinen and Markku Oivo. Experience with explicit modelling of relationship between process and product quality. In Proceedings of the Fourth Conference on Software Process Improvement (SPI ’98), Monte Carlo, Monaco, December 1998. Mark Paulk et al., “Capability Maturity Model for Software”, Version 1.1, CMU/SEI-93-TR-24, Feb. 1993. Frank van Latum, Rini van Solingen, Markku Oivo, Barbara Hoisl, Dieter Rombach, Guenther Ruhe, Adopting GQM-based Measurement in Industrial Environment, IEEE Software, 15 (1), January 19998, pp. 78–86. Andreas Birk, Pieter Derks, Dirk Hamann, Jorma Hirvensalo, Markku Oivo, Erik Rodenbach, Rini van Solingen, Jorma Taramaa: “Applications of Measurement in Product-Focused Process Improvement: A Comparative Industrial Case Study”. In Proceedings of the 5th International Symposium on Software Metrics (Metrics '98), IEEE Computer Society Press, Bethesda, Maryland, USA, November 1998 Andreas Birk and Felix Kröschel. “A knowledge management lifecycle for experience packages on software engineering technologies”. To appear in: Proceedings of the Workshop on Learning Software Organisations (LSO’99), Kaiserslautern, Germany, June 1999. ESPRIT Project PROFES. “The PROFES User Manual”. June 1999. Basili V. R., Caldiera G., and H.D. Rombach. Experience Factory. In John J. Marciniak, editor, Encyclopedia of Software Engineering, Vol. 1, pages 469-476. John Wiley & Sons, 1994.