guest editors' introduction The Case for ... - Semantic Scholar

focus 1

guest editors’ introduction The Case for Quantitative Process Management Bill Curtis, CAST Software

Girish V. Seshagiri, Advanced Information Services

Donald Reifer, Reifer Consultants

Iraj Hirmanpour, Software Engineering Institute

Gargi Keeni, Tata Consultancy Services

D

uring development, how can you determine with confidence whether your system will exceed its allowable downtime when delivered? When customers change their minds about functionality, how can you be confident that you’ve made the right tradeoffs to deliver the system within your original fixed budget? During the early weeks of a sprint, how can you be confident that you can deliver all the functionality described in the stories commit ted in the sprint? How can you determine when a system’s technical challenge has exceeded your team’s capability and will undermine your commitments for quality, functionality, cost, or schedule? During a project, how can you detect that the discipline required to perform some development practices has become lax? If you’re delivering lifecritical software, how do you know that you’re on track to have removed all the safety and securityrelated defects before delivery? Getting answers to these and other ques tions is often difficult because hard data is just not available. Even if it is, you still need to identify trends and isolate the root causes of problems. These types of questions drive organizations to embed statistical methods into their software engineering practices. They adopt these quantitative measures so that they can manage products and the pro cesses used to generate them.

First, the articles This special section of IEEE Software ex 24

IEEE Soft warE

plores the use of statistical methods em bedded into software engineering practice to support quantitative process manage ment (QPM). These methods complement others used for product management whose adoption is equally important. These processbased techniques aim to monitor development results in real time or to simulate development activities to support in situ decisions and actions re garding progress or quality. What dis tinguishes these methods is that we col lect data at the process event or task level rather than at the milestone or phaseend level. We can use these data to ■ evaluate the performance of process

tasks and their results, ■ gain insight into the causes of variation

in process performance, ■ analyze trends and pinpoint root causes

of problems, and

Published by the IEEE Computer Society

■ predict

future process or project outcomes.

Statistical methods have been used on software projects since the 1970s. However, most uses weren’t embedded in the process because the measures were aggregated at the end of a development phase to estimate fu ture results. Typical examples included the use of costestimating tools at the beginning of a project and the use of software reliabil ity models at the beginning of testing. Only recently have we begun to see sustained de ployment of statistical methods embedded in development processes. The two articles in this section come from companies in two industry segments, aerospace and IT outsourcing, where the use of statistical methods for QPM has grown fastest. These two segments have been particularly motivated to achieve CMMI’s higher maturity levels, in which quantitative 0 74 0 -74 5 9 / 0 8 / $ 2 5 . 0 0 © 2 0 0 8 I E E E

management of the development process is essential. CMMI embraces statistical meth ods in its quest to promote rigorous process improvement practices across an entire de velopment organization. You’re probably asking, what’s the value of embedding statistical methods into soft ware engineering practices? Even though such methods revolutionized the improve ment of quality and cycle time in manu facturing, can these methods yield similar results when used in creating intellectual rather than physical artifacts? The bene fits reported from highmaturity organiza tions suggest that they can,1 but they don’t provide definitive proof because isolating the impact of statistical methods from other improvements is difficult. However, the two articles in this section provide insight into how software organizations can gain value from embedded statistical methods. In the first article, Uma Sudhakar Rao, Srikanth Kestur, and Chinmay Pradhan at Unisys Global Services India demon strate how stochastic optimization model ing (SOM) can be embedded into the proj ect management process. They used SOM to select among alternative methods for implementing project processes, to iden tify processes to be statistically controlled, and to periodically recalibrate expectations about project results to guide corrective ac tions. Because this technique is sensitive to underlying assumptions about process performance, it highlights the importance of understanding distributions of task per formance results and grounding model as sumptions on historical process capability data. When the proper QPM foundation has been laid for SOM, it can become a powerful management tool guiding critical decisions and exploring whatif scenarios. In the second article, David Card, Kevin Domzalski, and Glyn Davies, using data from BAE Systems NS, demonstrate how embedded statistical methods can reduce delivered defect rates over time. Their mea surement system combines techniques such as control charts and profile matching. They show how to use a combination of control charts to identify process problems during inspections. They also show how to quan tify defect detection rates across the life cycle to develop and test predictions about quality results. There are several challenges

in using control charts with software data, and they provide grist for the debate pre sented in the Point/Counterpoint discussion in this issue.

Background More than helping attain a highmaturity CMMI level, these QPM methods’ real value is in providing what W.E. Deming called “a system of profound knowledge” to support inprocess decisions about de velopment.2 Embedded process measures have been used for decades to control and improve manufacturing processes. This practice has expanded into business pro cess management, where measures are now being embedded into endtoend business processes to manage the work flow’s effectiveness, the quality of its work products, and the delivery of value to the customer.3 The challenge in applying these techniques to software development is to adapt them from their original use with predominately routine tasks for applica tion to the knowledgeintense, nonroutine development of computer logic. In developing the field of statistical pro cess control, Walter A. Shewhart’s objective was “through the use of past experience” to “predict, at least within limits, how the phenomena may be expected to vary in the future.”4 Donald J. Wheeler reinforced this sentiment, maintaining that “prediction is the essence of doing business.”5 At the be ginning of this introduction, we listed some outcomes that executives, project manag ers, and developers would like to be able to

Developers’ ability to use their own performance data to manage and control their work processes provides a strong foundation for empowering them.

predict and control throughout a project, especially when safety, business, or reputa tion are at risk. Statistical methods can help these professionals achieve such results, es pecially when the added expense of achiev ing them is justified.

A QPM example Consider figure 1’s example of one form of QPM and the predictive use of process measures. Throughout a project, software managers and executives want accurate pre dictions of whether the project will meet its commitments for schedule, cost, features, quality, or other critical outcomes. There are many methods for developing these pre dictions; this example presents only one. The first step in providing these predic tions for each critical outcome is to develop a model of the project’s subprocesses or tasks that most affect each outcome, be cause the tasks having the greatest impact on each outcome might differ. Next, for each process task in the model, we define the measures of process performance and work product attributes that we believe are most predictive of final project outcomes, along with methods for analyzing them. We enable these two steps by having a standard software development process with associ ated measures. Consequently, QPM using inprocess measures is beyond the capabil ity of immature software organizations. The next step is to formulate an analy sis strategy supporting the predictions of the critical projectperformance param eters. In many cases, the predictive model will incorporate additional predictors as the project progresses. The predictive model is represented here as a general linear equa tion characteristic of regressionbased ap proaches, but other statistical techniques might be more appropriate depending on the predictive objective and nature of the data. Throughout the project, project man agers recompute outcome predictions with data emerging from the performance of re cent process tasks to determine whether the intermediate results indicate that the project will achieve its desired outcomes or whether corrective action is warranted. The validity of any predictive model de pends partly on the quality of the data en tered into it. So, we must ensure that the measurements taken from the performance May/June 2008 I E E E S o f t w a r E

25

Predictive model: �0 + �1X1

+

�2X2

+

�3X3

+

�4X4

+

�5X5

+

�

= Y^outcome

Task 1 result

Task 2 result

Task 3 result

Task 4 result

Task 5 result

Final outcome

Process task 1

Process task 2

Process task 3

Process task 4

Process task 5

Final product

Process performance task 1





Figure 1. One example of using quantitative process management techniques predictively.

of process tasks are valid indicators of ac tual process or product attributes. For in stance, as Card, Domzalski, and Davies demonstrate in their article, the number of defects found in a hastily performed peer review probably doesn’t accurately indicate the actual number of defects in the work product under review. Accordingly, to have faith that the re sulting measures are accurate predictors of project outcomes, developers must evaluate whether they performed their process tasks with appropriate discipline. Card and his colleagues evaluated the time spent in review preparation as an indicator that the process was under control and was being performed in a disciplined way. When review time was inadequate, inspection results were unre liable indicators of work product quality. Thus, process discipline is important be cause it not only improves task results but also produces more accurate predictors of project outcomes. Developers performing process tasks use their process performance data to evaluate and control the performance of their tasks. Measures of their task outcomes become leading indicators or intermediate predic tors of final project outcomes. The ability of developers to use their own performance data to manage and control their work pro cesses provides a strong foundation for em powering the development staff. Empower ing developers frees managers to focus more time on customers or strategic issues. Con sequently, a concomitant benefit of QPM is its contribution to an empowered culture. Rao, Kestur, and Pradhan demonstrate a 26

IEEE Soft ware

w w w. c o m p u t e r. o rg /s o f t w a re

different approach to predicting project out comes—by simulating the effects of project processes on a combination of project out comes. Initially, project managers can use such simulations to select the best compo sition of project subprocesses for optimiz ing several outcomes simultaneously. Dur ing the project, project managers can use the data emerging from the performance of process tasks to drive predictive simulations by assessing the probability that the project will achieve targeted outcomes on the basis of current performance. Regardless of the techniques selected, the analytics behind these predictive mod els rest on the statistical relationships de veloped from historical project data. When the historical data are insufficient for rigor ous statistical characterization, simulations can work from inferences about distribu tions and relationships among factors in the model. Even so, it’s critical that the organi zation constantly evaluate these models to ensure their predictive validity is sustained. Updating the models’ parameters when nec essary provides profound knowledge con cerning changes in the underlying condi tions affecting project performance.

Statistical methods and development paradigms Although this issue has no article on the Personal Software Process (PSP)6 or Team Software Process (TSP),7 these methods present uniquely powerful opportunities for embedding statistical methods into soft ware development practices. PSP and TSP apply statistics in a way that removes from

the data the greatest source of performance variation in software development: differ ences among individual developers and de velopment teams. When this source of varia tion doesn’t complicate the data, individuals and teams who study their results over time can gain profound knowledge of the causes of variation in their performance. As individual developers and their teams better understand their capabilities and the factors that affect them, they can ■ make

more accurate performance predictions, ■ control their performance in real time, ■ quickly identify the need for corrective action, and ■ provide continually updated predictions about task and project outcomes. Consequently, these methods provide a strong foundation for high-maturity prac tices. Such techniques could also make sub stantial contributions to the disciplined per formance of agile methods. The growing base of data from PSP and TSP implementations offers an excel lent research platform for investigating how individuals and teams can predict, control, and improve their performance. With sources of variation due to individu als and teams controlled or removed from the data, it’s far easier to study other fac tors that cause variation in software de velopment and develop statistical meth ods for predicting and managing them. Lessons learned in PSP and TSP research have implications for the performance

management and improvement of knowl edge-intense work in domains beyond software development. Neither do we have an article on agile methods.8 Some of the more advanced us ers of agile processes have embraced statisti cal methods in a quest to improve their pro cesses. By looking for root causes, they map trend data to their workflows in novel ways. Agile projects therefore provide yet another research platform for determining ways to predict performance as we try various soft ware development methods. By investigat ing sources of variation in the data, we can identify best processes and practices.

What’s needed to expand deployment? To accelerate QPM’s adoption and use in software development, we must take steps in five communities: executives and project managers, developers, system acquirers, pro cess improvement professionals, and soft ware engineering researchers.

Executives and project managers Too few executives and project manag ers have experience or training in using in-process data to manage work processes and predict project outcomes. Without this preparation, executives and managers are averse to risking their performance on seemingly esoteric methods. Six Sigma pro grams were more widely and successfully deployed than Total Quality Management programs because they insisted on extensive training of executives and managers and on establishing management alignment behind the program. Successful deployment of QPM meth ods and other high-maturity techniques in software will require this same concen trated effort to gain understanding and commitment. Executives’ interest will rise if they believe that these methods will pro vide predictive leading indicators early in a project and that these indicators can guide them in taking timely corrective actions or renegotiating commitments with the cus tomer. Project managers will appreciate that these methods provide more powerful management tools than the standard tech niques they learn while studying for a proj ect-manager certification. Agile project managers will benefit because their short

time scales require the earliest possible de tection of problems.

Developers Developers are rarely trained to measure their own performance and then use the data to control and improve their work. PSP and TSP have been rare exceptions in provid ing such training, and a growing number of schools are adopting these methods in their curricula. Developers also resist adopting such practices when the overhead of collect ing and using measures is high. Automated assistance is critical to sustained QPM. In most areas of human endeavor, the shorter the period in which a task or pro cess must be performed, the more impor tant the role of measurement and statistics in its performance and improvement. From this perspective, agile methods should be fertile ground for using statistical methods to estimate, predict, control, and improve performance at the individual and team lev els. In the more advanced agile shops, we frequently see graphs of test run results and story completion rates on the walls. Such data beg for statistical treatment to under stand and improve agile teams’ capabilities.

System acquirers Those who acquire software-intensive systems often have difficulty determining whether a system is on track to meet its cost, schedule, functionality, and quality targets. Data summarized at significant milestones often fail as leading indicators of problems. Acquisition managers whose suppliers can provide in-process statistics have better leading indicators and more insight into the results that their suppliers will ultimately deliver. Acquisition managers need greater understanding of the visibility they can gain from in-process statistics and how to embed such statistics in their acquisition methods.

Process improvement professionals The deployment of QPM in software proj ects requires support from one or more statistically savvy members of an improve ment or measurement team. People with the kind of training that Six Sigma Black Belts typically receive can support the statistical methods for evaluating performance on in dividual tasks. However, the development, validation, and recalibration of statistical

prediction models typically require more sophisticated statistical knowledge. For or ganizations to better understand what con trols their development performance, and for software engineering to become a more predictable discipline, people with such skills must become as indispensable as they are in business intelligence.3

Software engineering researchers The bar to adoption is high if software or ganizations must develop their own QPM techniques and models. The transition to these methods will occur much more rap idly if the research or entrepreneurial com munity can develop the underlying models and techniques and then help companies calibrate them to local conditions. This was the path that led to broad use of softwarecost-estimating models. However, research ers must still address several critical issues. Here are five such issues: ■ Process measurement. The available

measures are often the wrong measures for analyzing process performance at the task or event level. Measures defined for monthly reporting are often poor measures for analyzing process tasks. Also, different predictive models might require different operational definitions from the same area of measurement. For instance, system availability might be predicted more accurately by mean time to failure than by defect density so long as the test data fairly represent the opera tional environment. Identifying the most effective measurement definitions for different QPM objectives would save in dustry much trial-and-error frustration. ■ Statistical analysis of design work. Soft ware development is nonroutine, knowl edge-intense design work, from the ar chitecture phase through coding. Process events differ in terms of actors, compo nent difficulties, and conditions and are therefore typically affected by multiple sources of variation. Are statistical tech niques borrowed from manufacturing appropriate for characterizing this be havior, or do we need different statistical techniques for characterizing the capa bility and sources of variation in design work? The Point/Counterpoint essays associated with this special section (pp. May/June 2008 I E E E S o f t w a r e

27

About the Authors Bill Curtis is senior vice president and chief scientist of CAST Software, where he focuses on pre-

dicting long-term system quality and costs from statistical characterizations of system attributes and architecture. He’s also a coauthor of the CMM, People CMM, and Business Process Maturity Model. He holds a PhD from Texas Christian University and is an IEEE fellow. Contact him at [email protected].

Girish V. Seshagiri is CEO of Advanced Information Services. He’s a cofounder of the Watts

Humphrey Software Quality Institute in Chennai, India, and of three software process improvement networks. His interests are in business-goals-driven software process improvement and defect-free software delivery on time. He received his MSc in physics from the University of Madras and his MBA in marketing from Michigan State University. Contact him at [email protected].

Donald Reifer is president of RCI, a software firm specializing in metrics and management. His

professional interests focus on developing business cases for justifying changes in software organizations. He’s a senior member of the ACM and IEEE. He has an MS in operations research from the University of Southern California. Contact him at [email protected].

Iraj Hirmanpour is principal of AMS, a software process improvement firm. His research

interest is in model-based process improvement based on CMM, CMMI, TSP, and PSP. He holds a PhD in computer science from Florida Atlantic University. Contact him at [email protected].

Gargi Keeni is a vice president of Tata Consultancy Services. Her research interests include process

improvement, quality management systems, and business excellence. She’s a senior member of the IEEE, a member of the National Association of Software and Services Companies, and a member of the IEEE Software Advisory Board. She received her PhD in physics from Tohoku University, Japan. Contact her at [email protected].

48–51) debate the appropriateness of sta tistical process control methods drawn from manufacturing. These answers are critical for industrial practice. ■ Individual and team capability. In dividual and team performance is af fected by multiple sources of variation, some differing among successive tasks and others differing across projects. We need ways to represent how these sources affect capability and how to develop expectations for evaluating process performance as these sources of variation change. For instance, how should our expectations of process performance change on the basis of a developer’s experience or skill? How should we characterize a team’s capa 28

IEEE Soft ware

w w w. c o m p u t e r. o rg /s o f t w a re

bility on the basis of its mix of experi ence and skill levels? How do we adjust and apply QPM methods to get the best control and prediction in a mixedcapability environment? ■ Aggregating measures for system prediction. The example in figure 1 ap pears simple until we realize that we’re trying to predict a system characteristic using bundles of measures taken at the component level. For instance, we’re using defect data from inspections of components to predict system quality. When and how should we aggregate these predictors from the component level to predict a system attribute? ■ Validating benefits. QPM methods need stronger validation than the results of

fered in case studies, given that negative results are rarely published. We must test these techniques’ marginal value by comparing their additional contribu tions to prediction and control versus the value of statistical methods that are computed only at major milestones dur ing development. Validations of benefits in multiple industries help develop con fidence in managers who have little pre vious experience with such methods.

Q

pm in software engineering is still in its infancy. Many highmaturity organizations have used these methods to gain profound knowledge about their development capability and the factors that create variation in their per formance. Predicting project outcomes us ing in-process data is a nascent capability. However, with advances in statistical meth ods for software phenomena, early-phase prediction of effort, duration, functionality, quality, and other important project out comes could become commonplace. A common characteristic of engineering disciplines is their use of measures integrated into engineering work for analysis and de cision-making. Software engineering work hasn’t historically relied on embedded mea surement and analysis to the same extent as other engineering disciplines. If software en gineering is to be truly an engineering disci pline, techniques such as those demonstrated in this special section should become much more widespread on software projects. References 1. D.L. Gibson, D.R. Goldenson, and K. Kost, Performance Results of CMMI-Based Process Improvement, tech. report CMU/SEI-2006TR-004, Software Eng. Inst., Carnegie Mellon Univ., 2006. 2. W.E. Deming, The New Economics, 2nd ed., MIT Center for Educational Computing Initiatives, 1994. 3. T.H. Davenport and J.G. Harris, Competing on Analytics, Harvard Business School Press, 2007. 4. W.A. Shewhart, Economic Control of Quality of Manufactured Product, Van Nostrand, 1931. 5. D.J. Wheeler, Understanding Variation, SPC Press, 1993. 6. W.S. Humphrey, Introduction to the Personal Software Process, Addison-Wesley, 1997. 7. W.S. Humphrey, Introduction to the Team Software Process, Addison-Wesley, 2000. 8. A. Cockburn, Agile Software Development, Addison-Wesley, 2002.