COMPREHENSIVE APPROACH TO SOFTWARE MEASUREMENT Matias Vierimaa1 , Maarit Tihinen1 , Taito Kurvinen2 VTT Electronics1 , NetHawk 2
[email protected],
[email protected],
[email protected] ABSTRACT Measurement data collection, analysis, and utilisation within projects and organisation form a basis to monitor and improve software development. Utilisation of measurement data and the sharing of experiences throughout organisation have too often been overlooked by the industry. This is not due to lack of appreciation for data collection, analysis, and use, but rather to the fact that good practices, methods and tools for a more comprehensive approach have not been available. There are several good existing frameworks for software measurement, but they are often process improvement oriented and therefore do not consider the establishment of continuous long lasting measurements throughout the organisation and projects. Thus, when the improvement goals have been met, measurements are seldom continued, and valuable effort and time used to build measurement infrastructure and culture are not fully utilised. Furthermore, organisations and projects typically collect large amounts of data from software processes, but this data is often scattered in several locations. Therefore, an effective utilisation of existing data requires careful planning where the needs of both the organisation and projects are taken into account. Existing methods, such as Goal, Question and Metrics (GQM) or factor-metrics (IEEE Std. 1061-1992) are used in several organisations. The utilisation, however, is not systematic and the use of methods is seldom supported by a set of tools that would ease the use of software measurements in organisations. Thus, tool support and automatisation of software metrics collection, analysis and presentation would make the software measurements more attractive to software organisations with short time-to-market requirements and tight project schedules. VTT Electronics has applied software measurements in several process improvement projects with several national and international companies. This paper presents a measurement framework developed in the MIKKO project. This project is developing a comprehensive and tailorable measurement framework for the industrial software process, providing support for the data collection process and full utilisation of data at the project and organisational level. In the MIKKO project, the measurement framework is being used and evaluated in two case and five follow-up companies operating in telecommunications or embedded software area. The framework includes processes, methods, and tool support recommended in measurement definition, data collection, analysis and packaging. The paper is based on experiences gained during several years of work in software measurement programs. The paper focuses on requirements and main problems underlying, when a comprehensive measurement framework is built in organisations. The paper describes the main problems identified in software measurement with solutions that led to the establishment of a comprehensive framework with more detail. The paper also describes company specific requirements that were set for the software measurement framework during the MIKKO project. Requirements identified included
tailorability due to the constantly changing environment and different domains, and the ability to utilise metrics in an organisation and in projects to avoid data cemeteries. 1. INTRODUCTION The significance of software in industrial product development is constantly increasing. As there is a constant need to improve product quality and reduce lead-time, software has grown in size and complexity. Present tendencies towards diversification of software production also present a new challenge. As a result, industry needs more than ever to invest in software process improvement methods and in the systematic adoption of the best practices throughout company organisations. This development potential was clearly emphasised by a VTT Electronics (referred to hereafter as VTT) survey of the Finnish electronics industry in autumn 1998. Software measurement has a key role when the strengths and weaknesses of software development are made visible. If the companies are willing to improve their practices constantly, they need to use software measurements to follow up their software development. Software measurement has had a supportive role for a long time. It has been used together with the software process improvement and therefore its use as an independent approach has been limited. The increase in size and effort to build software has, however, made the quality of software engineering an important competitive factor. Software measurement is also changing from the supportive role into a more active one, therefore software measurement activities have to be carefully planned and implemented throughout company organisations and projects. Several approaches and measurement frameworks exist for variety of purposes, see for example [10]. Our aim is to establish organisation and project level measurements using currently existing best practices at a given environment. Our goal is not to reinvent the wheel, but instead to adapt current best practices for the needs of the companies. In this paper, we present a measurement framework developed in the MIKKO project. The MIKKO project is developing a comprehensive and tailorable measurement framework for the industrial software process, providing support for the data collection process and full utilisation of data at the project and organisational level. In the MIKKO project, the measurement framework is being used and evaluated in two case companies and in five follow-up companies. Figure 1 describes the MIKKO framework.
MIKKO Framework Process Process support
Method Method support
Tool Tool support
Process and product measurement Project Organizational Project Organizational Level level Level level Figure 1. MIKKO framework Following sections describe the problems in software measurement and the requirements that were set for the measurement framework. In addition, the measurement framework itself is described. 2. STARTING POINTS FOR SOFTWARE MEASUREMENT FRAMEWORK DEVELOPMENT Software measurement has been used and adapted in companies for several decades. The results, however, are mixed (see [2] and [3]). The following sections describe the problems related to software measurement. The problems initiated the establishment of a comprehensive approach to software measurement – the MIKKO framework. The problems that were identified during several years of software measurement and process improvement at VTT included: • When measurements are used as part of process improvement, continuous long lasting measurements are seldom considered. • There are good software measurement methods available, such as factor-metrics [4] and GQM [1], [11], but they are not systematically used and supported. • Measurements are seldom entirely supported by automatic tools, and even if tools exist, they are company specific. • There is often a lot of data available, but it is not used effectively. Thus the utilisation of measurements is not strong enough. We will now discuss each of these issues in more detail. 2.1 Measurement as part of software process improvement Several process improvement approaches, such as QIP, PROFES [9] and Pr2 imer [5], use software measurement mainly to evaluate the usefulness of software process improvement activities. This kind of use of software measurement is quite common, and it is also recommended to use software measurement together with software process
improvement. However, this kind of supportive role for software measurement has some drawbacks. Effort and time are always needed to establish and utilise software measurement. Software measurement requires planning, and measurement needs to be defined, analysed and utilised. In addition, software measurement requires a measurement culture. From a project viewpoint, measurement can be seen as a control mechanism and therefore the role of measurement must be explained, and equally, the usefulness of measurement should be made visible. If software measurement is used only for a short time to evaluate the results of software process improvement, software measurement effort and time have been wasted after the process improvement has been done. In addition, the usefulness of software measurement may not be visible enough and the measurement can be seen as an extra burden. Thus, the supportive role of software measurement is time and effort consuming. In addition, process improvements would benefit from a measurement infrastructure and culture that already exist in a company. If measurements are already in place, projects are familiar with it and possible additional measurements are not seen as a burden. 2.2 Systematic approach to software measurement There are several aspects of software measurement that need to be considered when software measurement is systematically used. Several key methods, such as GQM and factor-metrics describe how to use software measurements in companies. These methods offer solid techniques for establishing and using software measurement. However, when implementing software measurement in companies, persons responsible for measurement will face several challenges such as how to implement measurements in an effective and economical way at both project and organisation levels. Software developers need different kinds of measurements than department managers need, existing environment rarely supports measurement collection, analysis and reporting and so on. Instead of just adopting measurement methods, companies need to understand the role of software measurement in software development and to build up a measurement framework that fits into their needs. Thus, support for how to adapt existing methods based on company needs is needed. 2.3 Tool support for software measurement For a long time, tools have constituted a problem in software measurement because data recording systems are often company-specific, and it has been difficult to create a more general tool to support software measurements. Software measurement can utilise databases that contain information about various issues such as defects etc., project management tools, effort tracking systems and different specific software development tools including version management systems.
Software measurement would be even more effective if data collection and analysis were supported by tools [8]. It has been noted [6] that the automation of collection and the analysis of measurement data will bring significant benefits compared to the manual alternative. Motivation towards software measurement will be higher since tool support makes recording measurements less erroneous and automated collection of data decreases project members' work compared to the manual alternative. On the whole , automation makes the work easier in the analysis phase thus saving valuable time that projects can use to development work. However, automation of measurements is not always suitable. Measurements that are seldom collected or are not tool based might be easier to collect manually. Therefore, tool support for software measurement is recommended, but it should be planned with the overall situation in mind. It should also be noted that it is important to define and use good metrics - automation is not a goal in itself. 2.4 Utilisation of software measurement Companies that develop software today have a lot of data available about the software processes and products. Although the data is available, the effort to analyse and learn from data is often seen too time consuming due to the time-to-market pressure and changes in the environment. Therefore, measurement use should be planned, and systematic measurement utilisation should be one of the main goals of software measurement. There are several issues that make measurement utilisation difficult. Often data is scattered into several locations thus making the measurement utilisation laborious. Manual collection is often only an alternative, unless tool support for collection can be arranged as discussed in the previous section. Measurements at project and organisation levels are often seen as isolated parts. For example, test phase measurements may be seen as part of the software testing process, but collected measurements may not be used to support other measurements. Thus, when project level measurements are planned, a wider use of software measurements is seldom considered. 3. REQUIREMENTS FOR MEASUREMENT FRAMEWORK Based on the initial problems identified, software measurement framework requirements were studied in more detail. Requirements for the software measurement framework were set by the MIKKO project case companies, follow-up companies, and lessons learned from previous measurement projects. Requirements for the measurement framework included the following: Tailorability - Measurement framework should be tailorable so that it can be used in different software development environments. Especially, measurement processes vary from company to company, and therefore the framework should not be too strict. It should also be possible to use different methods during the process.
Learning - Measurement framework should support learning from measurements. Thus, the importance of the last phases of measurement (analysis, packaging) should be addressed. Measurements should provide added value to software development. Continuous measurement - Measurement framework should aim at continuous long lasting measurements. Measurements should be collected and analysed automatically where feasible, since manual effort reduces the motivation of using software measurement. Support - The framework should also act as a support mechanism to company specific measurement by offering solutions and supporting measurements by giving examples. Measurements are collected, or at least produced, by project personnel. If the project itself does not benefit from the measurement at all, a motivation to collect measurements will be low which results in inaccurate and erroneous data. Flexibility - The variety of measurement definition for different measurement purposes should be addressed. Measurement framework should be flexible to reflect the changing needs of the organisation and projects. Company and project specific needs should form a basis for measurement program implementation. 4. SOFTWARE MEASUREMENT IN NETHAWK The MIKKO project case company NetHawk operates in the telecommunications sector and is located in Oulu. NetHawk was founded in 1990 and it manufactures telecommunications testing and monitoring equipment like PC-based analysers, simulators and servers. Today, NetHawk employs 200 people and the products of the company are used in 40 countries. Software development projects mainly develop and maintain the Nethawk product family. NetHawk is a trusted partner among the leading telecommunications operators and manufacturers in Europe, Asia and North America. In NetHawk, measurement data has been collected since 1995 and the measurement practices are defined and adopted by project members. All projects collect a predefined set of metrics. The project plan template contains NetHawk's average values for the metrics to be collected. These average numbers are used in the review of the project plan. The table for the metrics results is maintained during the project and the final report presents the realised figures for the metrics. If the planned figures deviate from the average figures of NetHawk, the project manager has to describe and clarify reasons and details for this difference. NetHawk was interested in developing their metrics table and the results gathered from the past projects so that the forthcoming projects can use the measurement data even more efficiently to support planning, controlling and analysis of data during an ongoing project. At the organisation level, they were interested to obtain signals of the course of average values, for example at one-year intervals. The improvement needs focused on the utilisation of measurements both on the project and organisational level and on the automation of collection process. Although the significance of the project viewpoint is easy to understand, i.e. to get metrics to fit the needs of the project and support the quality goals and to function as a basis for decisionmaking, the viewpoint of the organisation is not forgotten. The organisation aspect
provides possibilities for improvements on the process level thus motivating project members to collect measurement data. 5. RESULTS In this section, we describe the main results of the MIKKO project, MIKKO measurement framework, automatic tool support for measurements, and of the results obtained so far from NetHawk. 5.1 MIKKO measurement framework MIKKO measurement framework is aimed to support software measurements at both the organisation and project levels. Measurement utilisation was seen as a key to successful software measurement. Therefore the MIKKO measurement framework describes the measurement process that provides support for measurement utilisation. In addition, MetriFlame tool [7], developed by VTT Electronics, supports the most critical aspects of software measurement by automating the measurement process from measurement collection to measurement analysis. Following Figure 2 describes the MIKKO measurement framework. The figure shows results from each phase.
Ch ara cte rise
Modelling results
it plo x E
Experience packages Current status
Feedback reports
Organisational and project needs
Analysis results
Me as ure me An nt aly pla sis n pla n
t en rem su ea ets rm t fo rics s lan t rge Ta Me tion p a ilis Ut
yse Anal
Defin e
MIKKO framework results
Implement
Figure 2. MIKKO measurement framework - results layer In Figure 2, white phases describe the activities at the organisation level, whereas grey phases describe activities at project level. The framework contains the phases that are
needed to implement the measurements at the organisation and project level. For each phase, methods and tools that can be used to support the phase are identified. When implemented in companies, methods should be adapted to the needs of the companies. The first phase of the selected approach is the Characterise phase. Its goal is to analyse the current status of software measurement process. In addition, company needs at organisation and project level are identified. The Define phase produces a target for measurement which identifies the primary aims of measurement. In addition, common metric sets should be produced. A utilisation plan is an important result and is a key issue to guarantee successful measurements. This plan contains information of how the measurements are planned to be used in a company at the organisation and project level. At the project level, measurements should support decision-making and offer status information about the project. On the organisation level, measurements are used, for example, to identify improvements, compare projects, and build prediction models. In the third phase, the selected measurements will be integrated to the projects. The work includes the implementation of selected methods and tools into existing projects. Measurement plan describes how the projects implement the measurements and the analysis plan describes how the measurements are analysed on the project level. Analysis of selected metrics is the fourth phase. This crucial phase includes the collection and analysis of measurement data in projects. This phase produces analysis results from the projects as well as feedback reports which can be used to identify needs and improvements for software measurement. The final stage of the measurement framework is the Exploit phase. In this phase results from projects are packaged into reusable components and utilised on the organisation level on the basis of the utilisation plan. The presented framework results can be produced by variety of methods including GQM and factor-metrics, for example. The framework describes a minimum set of results that should be produced no matter which methods are used. The MIKKO framework describes a new utilisation focused measurement framework. By providing the MIKKO framework to companies, it is possible to build and use software measurements that take, to more extent, into consideration how the software measurements are actually used in the company. 5.2 Automatic tool support In addition to the measurement framework, automatic tool support for software measurements has been created using the MetriFlame tool. The MetriFlame tool gives automatic support for the following tasks of software measurement process: • Data collection from data sources (including most common databases, Microsoft project etc.) at given intervals. • Repeatable data analysis based on internal data analysis techniques including basic statistical operations.
•
Presentation of the resulting data using internal format or standard formats such as JPG. Data source is first defined in MetriFlame, after which data is bound to specified analysis formula. After the data analysis and sources are known, presentations are created. After the user has described the above information (data sources, data analysis formulae, and data presentation parameters), MetriFlame will automatically produce the resulting presentations at selected intervals. This kind of approach, combined with measurement framework use makes it possible to solve several problematic issues in software measurements. 5.3 Results in NetHawk The MIKKO framework offers an upper level framework, under which it is possible to select methods and tools that are best suited for companies' needs. Based on the experiences so far, the selected approach offers solid support to software measurements. With companies, the use of MIKKO framework was started with the first phase, Characterise. It was carried out with both case and follow-up companies. First, important material such as process descriptions, project plans, quality plans, and final reports were collected. Secondly, based on the material collected, the first descriptions of the state of the measurements was created after which we interviewed the key persons of each organisation such as project managers, quality engineers, quality managers and so on. As a result, needs, strengths and weaknesses of the current measurement process became known. In the second phase, software measurements were defined based on the companies' needs that were made visible in the first phase. Since NetHawk's needs were largely related to the measurement analysis, only some of the defined metrics were adjusted. In the third phase in NetHawk, already established measurements were supported with a more detailed analysis of projects during the measurement collection. This analysis is based on the automatic tool support. To further support the analysis, several cost and effort estimation tools have been evaluated. Our goal was to build a comprehensive measurement framework and use it in the case companies on the basis of their needs. The selected solution, the MIKKO measurement framework has been found useful, and it offers the necessary flexibility which is suitable in serving companies with different and changing needs . The work in the MIKKO project is still on-going, but we have already found the presented approach useful in defining and using measurements from the organisation and project viewpoint. 6. CONCLUSIONS This paper has presented a comprehensive approach to software measurements on the organisation and project level. We have developed a systematic way of how software measurement should be implemented, by focusing on the utilisation of measurements and providing tool support for the most critical parts of software measurements. By using the software measurement framework, it is possible to use a variety of tools and
methods that best suit the company-specific needs, and establish and utilise software measurements that support companies in their software development. 7. ACKNOWLEDGEMENTS The MIKKO project is funded by the National Technology Agency of Finland (TEKES). The authors would like to thank Seija Komi-Sirviö, Jussi Ronkainen and Toni Sandelin from VTT Electronics for their valuable comments and input. 8. REFERENCES [1] Basili, V., Caldiera, G. and Rombach, H. 1994 "Goal Question Metric Paradigm”. In John J. Marciniak, editor, Encyclopaedia of Software Engineering, volume 1, pages 528–532. John Wiley & Sons, 1994. [2] Fenton N. and Pfleeger S. 1999 "Software Metrics: A Rigorous & Practical Approach", Second Edition, PWS Publishing Company, Boston, ISBN 0-53495425-1. [3] Hughes B. 2000 "Practical Software Measurement", McGraw-Hill. [4] IEEE Standards Collection, Software Engineering, USA, ISBN 1-55937-442-X, 1994. [5] Karjalainen J., Mäkäräinen M., Komi-Sirviö S., Seppänen V. Practical process improvement for embedded real-time software, Quality Engineering, 1996, Vol. 8, no 4, pp. 565-573. [6] Komi-Sirviö S., Parviainen P., Ronkainen J. 2000 "Measurement Automation: Methodological Background and Practical Solutions - A Multiple case study". The paper will be presented in Metrics 2001 Conference, April 4-6, 2001, in London. [7] MetriFlame www-pages., http://www.vtt.fi/ele/research/soh/products/metriflame/ index.html. [8] Parviainen, P., Järvinen, J. & Sandelin, T. Practical Experiences of Tool Support in a GQM-based Measurement Programme. In Hawkings, C., Ross, M., Staples, G. & Wickberg H. INSPIRE II, Process Improvement, Training and Teaching for the Future. The British Computer Society, 1997. Also in Software Quality Journal, Vol. 6, No. 4, 1997 [9] PROFES project, 2000 "PROFES Home Page", www.profes.org [10] Rombach, H., Basili, V., Selby R. (eds.) 1992 "Experimental software engineering issues: Critical assessment and future directions". International workshop, Dagstuhl Castle, Germany, September 1992, Springer-Verlag, Lecture Notes in Computer Science 706. [11] Solingen, R. and Berghout E., 1999 "The goal/question/metric method, a practical method for quality improvement of software development", McGraw-Hill, ISBN 007-709553-7, 1999.
