Software Process Improvement: A Road to Success Mahmood Niazi School of Computing and Mathematics, Keele University, ST5 5BG, UK
[email protected]
Abstract. Software process improvement (SPI) has received much attention in both academia and industry. SPI aims to improve the effectiveness of the software development process. Several different approaches have been developed for SPI, including the SEI’s Capability Maturity Model (CMM), more recently the Capability Maturity Model Integration (CMMI) and ISO's SPICE. Research shows that the effort put into these approaches can assist in producing high quality software. This paper has a two-fold objectives: first to review and summarise the empirical evidence thus far on the costs and benefits of SPI approaches; second to establish a relationship between different approaches to SPI and to seek and identify whether these approaches fulfil all the needs for an effective SPI initiative. The aim of this review is to analyse material about SPI approach and to set the scene for future research in the area of Software Process Improvement.
1 Introduction Problems associated with software quality are widely acknowledged to affect the development cost and time [1]. The annual Standish Group report showed that on average the percentage of software projects completed on-time and on-budget was only 34% in 2003 [2]. A recent study, conducted by a group of Fellows of the Royal Academy of Engineering and British Computer Society, shows that despite spending 22.6 billions pounds on IT projects in UK during 2003/2004, significant numbers of projects still fail to deliver key benefits on time and to target cost and specification [3]. There have been increasing calls for the software industry to find solutions to software quality problems [4]. Software organizations are realizing that one of their fundamental problems is to have an effective software development process [5; 6]. In order to have an effective software development process different methods have been developed, of which Software Process Improvement (SPI) is the one mostly used. The objective of this paper is to discuss and analyse different approaches to SPI in order to identify the issues that can undermine these approaches. The other objective is to analyse empirical evidence on the costs and benefits of SPI approaches. The overall aim is that SPI practitioners would utilise the results of this paper to support the business case for initiating SPI initiatives. To focus this study, I investigated the following research questions: RQ1. Does SPI really impact on organizations’ capabilities? RQ2. What is missing in current SPI approaches? J. Münch and M. Vierimaa (Eds.): PROFES 2006, LNCS 4034, pp. 395 – 401, 2006. © Springer-Verlag Berlin Heidelberg 2006
396
M. Niazi
The main purpose of addressing these research questions is to provide software practitioners with some insight into initiating SPI programmes. To answer these research questions, a literature review technique was adopted. Furthermore, results of the previously conducted empirical study were also used to answer these two research questions [7]. This two step process has given confidence that the findings of this paper are indeed important to provide some insights into initiating SPI initiatives.
2 Research Methodology The SPI literature consists of case studies, experience reports and high-level software process texts. Most of the studies describe real life experiences of SPI implementation and provide specific guidelines and recommendations for SPI implementation. I have analysed these published experience reports, case studies and articles in order to answer research questions described in Section 1. There were 3 categories of papers. Firstly, papers in which the authors have summarised the impact of SPI initiative on quality factors (time, cost, productivity and customer satisfaction). Secondly, papers in which SPI implementation was discussed but authors did not provide any summary of SPI impact on the quality factors. In this case, I have had to read each paper carefully to identify the SPI impact on any quality factor. Thirdly, I have also analysed a few papers in which the results of empirical studies were described. In order to reduce researcher’s bias I have conducted inter-rater reliability test in this process. Three research papers were selected at random and a colleague, who was not familiar with the issues being discussed, was asked to identify SPI impact on quality factors. In the end results were compared with previous results. In this paper a literature review technique was adopted in order to analyse SPI impact on cost, time, productivity and customer satisfaction. This means that a secondary source was used to analyse SPI impact of quality factors. The primary data used in the literature was not verified directly. Because this study is limited to the software industry, the gathered data reflects the perceptions of those individuals employed in this industry, and generalizations to other industries should be undertaken with extreme caution.
3 Does SPI Really Impact on Organizations’ Capabilities? In the previously conducted empirical study with 34 Australian practitioners [8], I asked SPI practitioners “Have SPI initiatives provided clear and expected benefits to the management?” Results show that 71% of the practitioners say that SPI initiatives provided clear and expected benefits to the management. Only 6% of the practitioners say SPI initiatives did not provide any benefits to the management. Conducting previous research has convinced me that the SPI impact on organizations capabilities would be best viewed in terms of cost, time, productivity and customer satisfaction.
Software Process Improvement: A Road to Success
397
3.1 Impact on Cost There are several accounts that describe the impact of an SPI initiative on project cost. A few of the key studies are described below: • Prior to the CMM initiative at Raytheon in 1988 [9] the average rework costs was about 41% of project costs. After CMM initiative, these costs dropped to about 20%. Fixing source code problems dropped by 80% and cost of retesting dropped by 50%. • The process improvement resulted Hughes annual savings about $2 million annually [10]. • Diaz and Sligo [11] have computed return on investment for Motorola’s process improvement initiatives, which is $611,200 for an $90,180 investment (i.e. total return of 677%). These results are very positive and show that efforts put into SPI can reduce overall development cost. 3.2 Impact on Time Many studies have described SPI impact on time. • Dion [12] described that before SPI initiative at Raytheon most projects were completed behind schedule and now most projects are finishing on schedule. • Herbsleb et al [13] described the yearly reduction in time to market in the range of 15% to 23% (i.e. 19% Median). • Herbsleb and Goldenson’s [14] found a correlation between higher maturity and meeting schedules. For example, ability to meet schedule increased from 40% (companies in CMM Level 1) to 80% (companies in CMM Level 3). • These results show that the SPI is an effective approach that can help organizations in reducing time to market. 3.3 Impact on Productivity The following studies have highlighted SPI impact on productivity: • Herbsleb and Goldenson’s [14] results showed development productivity of companies has increased from 50% (companies in CMM Level 1) to 90% (companies in CMM Level 3). • Herbsleb et al [13] described the yearly gain in productivity in the range of 6% to 67% (i.e. 35% Median). • SPI initiative at Raytheon obtained an average productivity increase of about 130% from 1988 to 1992 [12]. • Butler [15] described a most recent development project 10 time more productive than the baseline project. The results show that organizations productivity has been increased with the passage of time after adopting SPI initiatives.
398
M. Niazi
3.4 Impact on Customer Satisfaction There are several studies that describe SPI impact on customer satisfaction. A few of the key studies are described below: • Yamamura [6] results showed a correlation of employee satisfaction to process maturity, e.g. the average satisfaction was increased from 57% (before process improvement activities) to 83% (after process improvement activities). • Herbsleb and Goldensons [14] showed that customer satisfaction has increased from 80% (companies in CMM Level 1) to 100% (companies in CMM Level 3). • Pitterman [5] reported an overall customer satisfaction increased at Telcordia Technologies from 60% in 1992 to above 95% in 2000. The results show that increased organizations’ process maturity has also increased customer satisfaction with the products. 3.5 Discussion The analysis of the literature shows that large enterprises using processes based on SPI models and standards can produce higher quality software, reduce development cost and time, and increase development productivity. However, most research on the impact and benefits of SPI approach has focused on large organizations [11; 16]. There is a world-wide academic and industrial interest in similar research on Small and Medium-sized Enterprises (SMEs). Research shows that SPI models like CMMI is difficult to apply to (SMEs), due to distinguishing characteristics of SMEs [17]. Due to these distinguishing characteristics, it is important to analyse the impact of SPI approach on SMEs. This is because SMEs may need to know that SPI is proven before they may be drawn to complex issues of SPI which require lot of funds, expertise and management time.
4 What Is Missing in Current Approaches to SPI? Different models and standards have been developed in order to improve software processes. The CMM is developed by software engineering institute (SEI) in order to improve organizations’ software processes. The Capability Maturity Model Integration (CMMI) [18] is the latest SPI model from the SEI. SPICE is a set of international standards for software process assessment [19]. SPICE is intended to harmonize many different approaches to software process assessment and to provide an approach that encourages self-assessment. The ISO 9000 series of standards [20] were developed with the intent of creating a set of common standards for quality management and quality assurance. 4.1 Discussion In order to address the effective management of software process different approaches have been developed, of which SPI is the one most often used. Research shows that the effort put into these model and standards can assist in producing high quality software [6; 21].
Software Process Improvement: A Road to Success
399
Despite these documented benefits, SPI initiatives exhibit low levels of adoption and limited success [22]. Deployment is often not only multi-project, but multi-site and multi-customer type and the whole SPI initiative typically requires a long-term approach. It takes significant time to fully implement an SPI initiative [23]. Such time frames mean that the SPI approach is often considered an expensive approach for many organizations [22] as they need to commit significant resources over an extensive period of time. Even organizations willing to commit the resources and time do not always achieve their desired results. The failure rate of SPI initiatives is very high, estimated as 70% [24; 25]. The significant investment and limited success are reasons for many organizations being reluctant to embark on a long path of systematic process improvement. Despite the importance of SPI implementation process, little empirical research has been carried out on developing ways in which to effectively implement SPI programmes [16; 22]. Much attention has been paid to developing standards and models for SPI. This suggests that the current problems with SPI are not a lack of standards or models, but rather a lack of an effective strategy to successfully implement these standards or models. A thorough literature review [11; 14; 21; 26; 27] and previously conducted interviews with 34 Australian practitioners [7] revealed that in general no standard approach has been adopted by practitioners for the implementation of SPI initiatives. Organizations typically adopt ad hoc methods instead of standard, systematic and rigorous methods in order to implement SPI initiatives [28]. So far no approach has been identified that could assist specifically in the design of effective SPI implementation initiatives. There is a great need to develop some mechanism that could assist SPI practitioners in the design and implementation of effective SPI initiatives. This has the potential to reduce SPI implementation time, cost and failure risks. In order to address some of the missing SPI issues a research project with SPI practitioners is being carried out [7]. The objective of this project is to develop SPI implementation framework to assist SPI practitioners in the design of effective SPI implementation initiatives. The results will be published as soon as the evaluation is completed.
5 Conclusion This paper has examined the domain of SPI. I have set out to answer two research questions in this paper: • RQ1. Does SPI really impact on organizations’ capabilities? SPI literature shows that the SPI approach can help organizations in reducing development cost and in improving time-to-market, productivity, and customer satisfaction. However, in order to support the business case for initiating SPI initiatives in SMEs, it is important to analyse the impact of SPI approach on SMEs. • RQ2. What is missing in current SPI approaches? Thorough literature review revealed one topic of SPI missing, i.e. many standards and models exist for SPI but little attention has been paid to their effective implementation. Literature also shows that in real life different organizations adopted
400
M. Niazi
chaotic methods instead of standard methods in order to implement SPI initiatives. So far no approach has been identified that could assist specifically in the design of effective SPI implementation initiatives. Therefore, further research is needed in the domain of SPI implementation in order to reduce SPI implementation time and cost, higher SPI implementation quality and higher SPI practitioners’ satisfaction.
References 1. Sommerville, I.: Software Engineering Fifth Edition. Addison-Wesley.(1996). 2. Standish-Group: Chaos - the state of the software industry. (2003). 3. The-Royal-Academy-of-Engineering: The Challenges of Complex IT Projects, The report of a working group from The Royal Academy of Engineering and The British Computer Society. ISBN 1-903496-15-2 (2004). 4. Crosby, P.: Philip Crosby's reflections on quality. McGraw-Hill.(1996). 5. Pitterman, B.: Telcordia Technologies: The journey to high maturity, IEEE Software (July/August). (2000) 89-96. 6. Yamamura, G.: Software process satisfied employees, IEEE Software (September/October). (1999) 83-85. 7. Niazi, M., Wilson, D. and Zowghi, D.: A Framework for Assisting the Design of Effective Software Process Improvement Implementation Strategies, Journal of Systems and Software Vol 78 (2). (2005) 204-222. 8. Niazi, M., Wilson, D. and Zowghi, D.: A Maturity Model for the Implementation of Software Process Improvement: An empirical study, Journal of Systems and Software 74 (2). (2005) 155-172. 9. Hally, T.: Software process improvement at Raytheon, IEEE Software November (1996) 10. Humphrey, W. S., Synder, T. R. and Willis, R. R.: Software process improvement at Hughes Aircraft, IEEE Software 8 (4). (1991) 11-23. 11. Diaz, M. and Sligo, J.: How Software Process Improvement helped Motorola, IEEE software 14 (5). (1997) 75-81. 12. Dion, R.: Process improvement and the corporate balance sheet, IEEE Software 10 (4). (1993) 28-35. 13. Herbsleb, J., Caarleton, A., Rozum, J., Siegel, J. and Zubrow, D.: Benefits of CMM-based software process improvement: Initial results. Technical report, CMU/SEI-94-TR-013 (1994). 14. Herbsleb, J. D. and Goldenson, D. R.: A systematic survey of CMM experience and results. 18th international conference on software engineering (ICSE-18). Germany (1996) 323-330. 15. Butler, K.: The economics benefits of software process improvement, CrossTalk (July). (1995) 14-17. 16. Goldenson, D. R. and Herbsleb, J. D.: After the appraisal: A systematic survey of Process Improvement, Its benefits, And Factors That Influence Success. SEI, CMU/SEI-95-TR009 (1995). 17. Cater-Steel, A.: Process Improvement in Four Small Software Companies. 13th Australian Software Engineering Conference (ASWEC'01). (2001) 18. SEI: Capability Maturity Model® Integration (CMMISM), Version 1.1. SEI, CMU/SEI2002-TR-029 (2002). 19. ISO/IEC-15504: Information technology - Software process assessment. Technical report - Type 2 (1998). 20. ISO-9000: Internation Standard Organization, http://www.iso.ch/iso/en/iso900014000/iso9000/iso9000index.html, Site visited 23-02-2004. (2004).
Software Process Improvement: A Road to Success
401
21. Ashrafi, N.: The impact of software process improvement on quality: in theory and practice, Information & Management 40 (7). (2003) 677-690. 22. Leung, H.: Slow change of information system development practice, Software quality journal 8 (3). (1999) 197-210. 23. SEI: Process Maturity Profile. Software Engineering Institute Carnegie Mellon University, (2004). 24. SEI: Process maturity profile of the software community. Software Engineering Institute, (2002). 25. Ngwenyama, O. and Nielsen, P., A.: Competing values in software process improvement: An assumption analysis of CMM from an organizational culture perspective, IEEE Transactions on Software Engineering 50 (2003) 100-112. 26. Butler, K.: Process lessons learned while reaching Level 4, CrossTalk (May). (1997) 1-4. 27. Dyba, T.: Factors of software process improvement success in small and large organizations: an empirical study in the scandinavian context, ACM SIGSOFT Software Engineering Notes 28 (5). (2003) 148-157. 28. Zahran, S.: Software process improvement - practical guidelines for business success. Addison-Wesley.(1998).
