Measuring Process Consistency: Implications for Reducing Software ...

800

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 25,

NO. 6,

NOVEMBER/DECEMBER 1999

Measuring Process Consistency: Implications for Reducing Software Defects M.S. Krishnan and Marc I. Kellner AbstractÐIn this paper, an empirical study that links software process consistency with product defects is reported. Various measurement issues such as validity, reliability, and other challenges in measuring process consistency at the project level are discussed. A measurement scale for software process consistency is introduced. An empirical study that uses this scale to measure consistency in achieving the CMM goal questions in various key process areas (KPAs) in 45 projects at a leading software vendor is reported. The results of this analysis indicate that consistent adoption of practices specified in the CMM is associated with a lower number of defects. Even a relatively modest improvement in the consistency of implementing these practices is associated with a significant reduction in field defects. Index TermsÐSoftware process consistency, software process measurement, CMM, software defects, empirical model.

æ 1

S

INTRODUCTION

OFTWARE development and maintenance of existing software is a growing component of our economy. Global software expenditures exceed several billion dollars [33]. Quality and timely delivery of software within reasonable costs are critical for the success of many business organizations today. Yet, the performance of the software engineering field has not lived up to the dynamic changes in technology and requirements encountered by software organizations. For example, a recent survey article has reported that software failure costs are approximately 20 percent of the revenue for firms with incomes up to $10 million [31]. It is well recognized that developing large-scale software systems and the coordination of various tasks involved in this development is a complex undertaking. This complexity is one of the main reasons for cost and schedule overruns, poor quality, and other problems frequently associated with software. Software engineering solutions aim to manage this complexity by employing sophisticated tools, representation techniques (of user requirements, architectures, designs, etc.), programming languages, and approaches, methods, and practices followed in developing the software, management techniques, and so forth. One of the major solutions, recently proposed for avoiding and managing the problems frequently associated with software development, is to treat the entire development task as a process that can be measured, controlled, and improved [28]. This approach helps to force discipline into the methods and practices adopted during software development. In general, an organizational process can be defined as a logical organization of people, technology,

. M.S. Krishnan is with the University of Michigan Business School, Ann Arbor, MI 48109. E-mail: [email protected]. . M.I. Kellner is with the Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213. E-mail: [email protected]. Manuscript received 15 Sept. 1997; revised 25 July 1998. Recommended for acceptance by C. Ghezzi and B. Nuseibeh. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number 109061.

and practices into work activities designed to transform information, materials, and energy into a specified end result (adapted from [45]).A software process can be defined as a set of partially ordered activities performed during the development or evolution of a software product or system [12]. An important stream of research in process focuses on frameworks for software organizations to use in characterizing the state of their current software practice in consistent terms and in setting goals and priorities for improving their processes. One widely adopted example is R 1 ) framethe Software Capability Maturity Model SM (CMM work developed by the Software Engineering Institute (SEI) at Carnegie Mellon University [47]. The CMM specifies 18 key process areas (KPAs) classified into five process maturity levels, as illustrated in Fig. 1. The basic premise is that a higher maturity level process will lead to more effective results in terms of increased productivity and reduced defects and cycle-time. Each of these process areas in the CMM includes specific software practices to be followed by the organization. The CMM framework aids software organizations to assess their own process capabilities and identify the most important areas of improvement. In this paper, we study the link between projects' consistent adoption of the practices specified in the CMM and the field defects reported in the resulting software products. In a practical setting, due to various reasons such as schedule pressure and human error, the practices defined in the CMM may not always be followed in a project even if the intention was to follow them carefully. Serious consequences, in terms of project and product characteristics, may result from such inconsistent implementation of good practices. For example, in the project tracking and oversight KPA, the CMM requires software organizations to periodically compare a project's actual results (e.g., cost, schedule, and size) with the initial 1. Capability Maturity Model is a service mark of Carnegie Mellon University. CMM is registered in the U.S. Patent and Trademark Office.

0098-5589/99$10.00/99/$10.00 ß 1999 IEEE

KRISHNAN AND KELLNER: MEASURING PROCESS CONSISTENCY: IMPLICATIONS FOR REDUCING SOFTWARE DEFECTS

801

Fig. 1. Overview of the SW-CMM. (This figure courtesy of Mark C. Paulk of the Software Engineering Institute at Carnegie Mellon University.)

software plans and take corrective actions when there is a mismatch. If this practice is not regularly followed, it may give rise to serious deviations between the project estimates and the actual final results; unfortunately, cost and schedule deviations in excess of 100 percent are not uncommon in software projects in low process maturity settings. Thus, consistently implementing the practices specified in the CMM is expected to improve project and product outcomes. This paper specifically investigates the benefits in product quality (i.e., lower number of defects) associated with a reduction in process inconsistencies with respect to the CMM. The empirical study reported here is based on data collected on 45 products from a leading software vendor. Measurement issues in software process, such as validity and reliability, are discussed and consistent adoption of various practices specified in the CMM is measured for the projects in our sample based on a frequency scale. Our measures of software process at a project level using this scale exhibit reasonable reliability coefficients. Our findings indicate that consistent adoption of the CMM practices in a software project is associated with a lower number of defects (i.e., defects discovered during customer use per thousand lines of code). The rest of the article is organized in the following way. Section 2 addresses a collection of areas of related research into process consistency and improvement and their relevance to this work. Section 3 introduces our research hypothesis and outlines our empirical model and approach. Section 4 discusses various challenges in measuring software processes and presents a new scale for measuring consistent adoption of the CMM practices. Details of the data, analysis, and model results are discussed in Section 5.

Section 6 is the conclusion and suggests directions for future research.

2

BACKGROUND

2.1 Major Streams of Process Research Software development involves interactions among people, technology, methods, and procedures. This collective interaction viewed as a sequence of activities is termed a software process [28]. Research and development work in the arena of software process has led to a number of models, frameworks, and technologies, for defining, assessing, and improving the software processes followed in an organization. These developments can be largely classified into three major streams of research, as follows. The first stream of research in process focuses on frameworks for software organizations to use in characterizing the state of their current software practices in consistent terms and in setting goals and priorities for improving their processes. In the past decade a number of methods and frameworks for assessing software process maturity or software process capability of organizations have been developed. The CMM [47] introduced in Section 1 and its associated process appraisal techniques (along with the work that led up to these [28]) are a very well-known example of work in this stream. Some of the other frameworks for software process improvement are SPICE, ISO-9000-3, Bootstrap, and Trillium [16], [46], [41], [24]. Most of the work in this stream is based on fundamental principles of quality management. The basic premise of these frameworks is that consistent application of welldefined and measured software processes, coupled with continuous process improvement, will substantially im-

802


prove the productivity of software organizations and the quality of their products. The second stream focuses on measurement-based improvement approaches. This includes the large body of work done at the NASA Goddard Software Engineering Laboratory (SEL) [2], [3]. Their quality improvement paradigm (QIP) focuses on developing and identifying best practices within the organization, without heavy reliance on industry-wide best practice models and frameworks (which are the focus of the first stream of research). QIP involves experimentation with new practices and technologies in the target organization, measurement of results, and quantitative analysis of those outcomes to determine best practices and circumstances for their useful application. The Personal Software Process SM (PSP SM )2 approach is another example of work in this stream and helps individual engineers improve their software practices through defined processes, measurement of individual results, statistical process control techniques, etc. [29]. The third stream involves methods and technologies that facilitate formal process definition and representation, analysis, and automation. Considerable research has been performed in the development and application of formal and semiformal process modeling languages to support these varied objectives [12], [19], [26]. Powerful process analyses, both qualitative and quantitative, are supported by techniques such as process model simulation [49]. Workflow Management Systems and Process-Centered Software Engineering Environment (PSEE) tools aim at supporting the set of activities involved to conceive, design, develop, and maintain the software products [6], [14]. They require formal representations of software processes in an explicit manner through process modeling languages. The automation tools use these process models to guide and support software engineers taking part in the process and aid in automating the execution of various process steps. However, as reported in [14], one of the challenges in building and using process automation technology is its inability to tolerate unexpected situations and deviations from the specified process models. Such deviations from the defined process models often lead to inconsistencies between the actual process and the corresponding process support systems represented by the technology. Although these three streams of process research have for the most part developed independently, there have also been notable cases of cross-fertilization and integration across them. For example, the PSP approach [29] is explicitly related to the CMM framework [47]. As a second example, some of the process modeling technology has also been oriented at supporting process improvement along the lines suggested in the CMM framework [12], [34].

2.2 Research Issues in Process Consistency In order to ensure the quality of the delivered products, consistency, and effectiveness of the software process is required. Within the technology (third) stream of process research discussed above (Section 2.1), several process automation tools have been introduced to improve and support software development [6], [14], [19]. Certain issues of process consistency have arisen related to this technology. 2. Personal Software Process and PSP are service marks of Carnegie Mellon University.

VOL. 25,

NO. 6,


As an attempt to introduce discipline and improve consistency in software processes, solutions based on process automation support technology have prescribed formal methods to enforce rigid constraints such that software engineers may not be able to perform an operation unless it has been anticipated and described in a defined process model. However, this approach has not been successful due to the following reasons. First, in a software development project, it may not be possible to anticipate all the operations and activities along with the permutations of their sequences. Second, since these solutions impose predefined behavior, this may restrict human creativity, that is critical to project success and process improvements. Third, determined software engineers can and will circumvent any restrictions imposed by a tool, if they choose to do so. These limitations have been recognized and recent methods and tools in software process modeling have adopted a more dynamic approach to account for unexpected deviations and the resulting inconsistencies between the real-world process and the process model used by the tool [14]. For example, SENTINEL, a process centered software engineering tool, detects and allows users to describe temporary deviations from the process model [13]. SPADE is another process centered software engineering tool that allows dynamic changes of the process models. That is, based on information on unexpected situations, users can update the process definitions and SPADE will consistently update its internal states accordingly and resolve inconsistencies [1]. A comprehensive analysis of the deviations and resulting inconsistencies need a cognitive understanding of the behavior of the humans performing the process. Cugola et al. [14], classify process deviations and the resulting inconsistencies into the following two levels: 1) domain and 2) environment. Deviations occurring within the software development process are called domain-level deviations. That is, an event or a sequence of tasks in software development that deviates from the expected behavior of the process model defined within the process support tool. For example, suppose a software developer writes the unit test results report before writing the test plan. If the defined process prescribes that the test plans must precede test results, then this brings about an inconsistent state. These inconsistencies are referred to as domain level inconsistencies. Formal process technology solutions have been proposed to describe domain level inconsistencies in a process model [19]. Environmental level deviations occur due to the differences between the actual software development and the process defined in the process support system. In this deviation class, the event performed by the software engineers in the project is not mirrored by a corresponding event in the process defined in the process support tools. In other words, the process automation support system fails to capture these unexpected events. The process inconsistencies arising from such deviations are termed environmental deviations. In this article, we do not address the issues related to representing deviations or inconsistencies in a process nor


the issues involved in bringing a process automation tool and its process model back into a state consistent with the real-world process being performed. Rather, we argue that both domain and environment level process deviations may be introduced into a process when project managers or developers fail to consistently follow the practices prescribed in rigorous software process frameworks such as the CMM. For example, within the peer reviews KPA, the CMM prescribes that the actions associated with the defects that are identified during the peer reviews must be tracked until they are resolved. If this practice is not followed consistently in a software project and the defined process prescribes creation of a defect action resolution report, then a domain level process deviation is introduced. Environmental deviations may also be caused by not following the practices prescribed in the CMM. For example, within the process change mnagement KPA, the CMM prescribes that improvement and changes made in the software development process must be continually updated in the organization's defined software process. If this practice is not followed in a software project, then it may soon lead to a task in software development that is not mirrored in the defined software process, creating an environmental process deviation. Thus, we have explained that process automation tools have not been successful in forcing consistency with a prescribed process. It appears that engineering and managerial discipline are needed to ensure consistent adoption of the CMM practices (or other good practices). Moreover, we have shown that failure to consistently apply the practices described in software process frameworks, such as the CMM, can easily lead to process deviations and the resulting challenges they pose for process automation tools. As discussed above, other work is addressing these automation issues. On the other hand, this article focuses on the potential benefits to be gained (in a business value sense) by avoiding process inconsistencies with respect to the CMM.

2.3

Business Value of Software Process Improvement An issue that potentially cuts across the three streams of process research (discussed in Section 2.1) focuses on the business value of the practical application of the methods, frameworks, technology, and so forth. While this issue is certainly applicable to the technology (third) stream, it has gone largely unaddressed for process modeling and automation technology. On the other hand, business value measurement is an inherent component of work in the measurement-based improvement (second) stream. However, the results of applying QIP are primarily focused on the one organizational setting of the SEL [3]. Business value results of applying the PSP approach are still emerging [25]. Most of the work on business value has concentrated on the assessment and improvement framework (first) stream of process research described above. Within this stream, the research effort to date has focused largely on developing these frameworks and associated methods. In parallel, howeverk, questions have been raised regarding the business value of adopting the practices specified in the CMM or other process frameworks and improving process

803

consistency in those terms. Software managers are understandably interested in assessing the return on investment from process improvement in their respective projects. Recent reports on business values of software process improvement efforts include case studies and survey-based research that provide evidence for improvements in product quality and productivity [20], [27]. Cook, Votta, and Wolf [9] report a case study that analyzes an in-place industrial process and show the linkage between specific process features and product outcomes. Krishnan [38] conducted a field study to assess the business value of process improvements measured in terms of increases in quality and productivity. This was one of the initial efforts to link process improvements to project performance in a rigorous empirical study. In a recent version of the traditional software cost model COCOMO, a process measure based on CMM assessment scores has been identified as one of the cost drivers in software projects [7]. Gopal et al. [23] studied the effect of process improvements in offshore software projects. There is a need for further empirical research, involving several projects, that links software process measurements and process consistency to quality or productivity outcomes. There are several challenges in performing such research. This article focuses on the potential benefits to be gained by avoiding process inconsistencies with respect to the CMM. In particular, we focus on the benefits in terms of the quality of the software products produced by these processes. In the remainder of this paper, process inconsistency means inconsistency with respect to the CMM. We next discuss our hypothesis and empirical model and then address the challenges in measuring the consistency of adoption of the CMM practices in a software project.

3

EMPIRICAL MODEL FOR SOFTWARE DEFECTS

The primary focus of this study is to investigate the effect of software process practices specified in the CMM on the number of defects in the resulting products. Several factors determine the number of defects in a software project. These factors may be classified into product, process, and people related. In product factors, both size and nature of the software product are believed to influence software defects [8], [43]. Since larger software products may contain more modules and interactions between them, the number of field defects is expected to be significantly higher. However, an interesting question is whether the rate of increase in defects in larger products is higher than the rate of increase in size. We investigate this question by normalizing defects by including product size as an explanatory variable in our model. In this study, all the projects are systems software products and, hence, significant differences due to the type of the software is not expected here. The quality of the personnel in a software team is also believed to significantly improve both productivity and number of defects reported in a software project [48], [40]. Turning to process factors, the effect on defects of adopting the CMM process improvement practices has been reported recently in the literature on business value of process improvement [20], [27]. These reports are based on

804


longitudinal case studies of a few projects within an organization, overall results for an organization taken as a single unit, or responses to process improvement surveys. An important difference between our study and these survey based findings is that in our study we use an objective dependent variable based on the number of defects, whereas previous studies have mostly used perceptions of product quality. As noted earlier the focus in this article is on studying the effect of consistent adoption of the CMM software practices on the number of defects in the resulting products. Our approach is to perform a cross-sectional study of numerous products within a single organization. We control for the effects of product and people factors, so as to tease out the specific benefit attributable to process consistency. Hence, we model the number of software defects as a function of product size, consistent adoption of the CMM practices, and personnel capability of the team, as defined in (1). NumberÿofÿDefects F unctionProduct Size; Consistent Adoption of

1

CMM P ractices; P ersonnel Capability Of course, we hypothesize that consistent adoption of the CMM practices will be associated with lower number of defects. In the remainder of this article, we discuss issues in measuring our independent variable (for process), detailed definitions of our variables, our data, and corresponding results.

4

SOFTWARE PROCESS MEASUREMENT

Two main challenges are faced by researchers in implementing a process value based study such as this one. These are discussed next. First, the maturity assessment of software process is often conducted at an organizational level. One may attempt to study the effect of process across a number of organizations. But this usually limits the volume of data that becomes available to a researcher. In addition, objective definitions of business value outcomes such as quality and productivity may not be uniformly defined across several organizations; these measures are often aggregated across different projects within the organization. Hence, projectspecific information is missed. Moreover, a cross-sectional study of process impacts across organizations may also need controls for a number of organizational specific effects such as organization size, business, and application areas, markets, etc. Hence in this study the approach taken is to investigate the effect of process across various projects within a single organization. The second challenge in conducting an empirical analysis of process value is related to errors in process measurements. El Emam and Madhavji [17] recently addressed the measurement issues in evaluating software process. They illustrated techniques for estimating validity and reliability of process maturity assessments and tested hypotheses that link organizational maturity with the effectiveness of requirements engineering. One source of measurement errors in process assessment could be the

VOL. 25,

NO. 6,


measurement scale used. Hence, in order to improve validity and reliability, there is a need for suitable definition of measurement scales in software process. A new frequency scale to measure adoption of software practices is introduced next and the validity and reliability of the data collected using this scale are addressed.

4.1 Measurement Scale for CMM KPA Questions The CMM specifies five maturity levels to assess an organization's process capability by measuring the degree to which processes are defined and managed. Each maturity level consists of several key process areas (KPAs) [47], (see Fig. 1). The KPA details are classified into goals and common features. In each KPA, several goals are defined, and under common features, certain key practices are defined. The CMM questionnaire contains several questions related to the goals and common features for each KPA [52]. The goal questions are aimed at measuring the implementation of the goals of the KPA, whereas the common features questions check the institutionalization of the KPA goals. In the CMM questionnaire, responses to these questions are measured on a binary ªYESº/ºNOº scale [52]. We conducted an initial pilot study of process conformance measurements with five product managers at our research site using the CMM questionnaire that requires the respondents to answer the KPA goal questions on a ªYESº/ªNOº binary scale. The primary difficulty was due to the inconsistency in adopting the methods and practices addressed by the CMM KPA goal questions. It was learned that the project managers at the research site were not consistent in following these practices in a software project. That is, managers followed the practices sometimes during the project, but not on all the opportunities in a project. For example, defect prevention practices were followed sometimes, but not in all the opportunities, for defect prevention during the project. Hence, it was difficult for the managers to respond on a strict Yes/No scale. A further discussion with researchers at the SEI revealed that this issue was also raised in the pilot field trials of Interim Profile measurements, a method developed by the SEI for rapidly assessing software engineering maturity status [51]. Another reason why we did not use the ªYESº/ªNOº response is because it did not fit our hypothesis. Our hypothesis is that value from implementing CMM KPA goals is not a discrete step function such that there is no effect untill the goals are fully met. We believe that the business value from CMM is less discrete and even partial adoption of the KPA goals may reduce defects. The significant gap between ªYESº and ªNOº responses to CMM KPA questions has been addressed earlier [22]. Goldenson [22], proposes a four point scale for response to each KPA question with the anchors defined as Yes, At times, Planned, and No. This scale helps in reducing the gap between Yes and No in response to the KPA questions. Since the focus of this study is to link consistent implementation of KPAs in projects to the product defects, including a ªplannedº status of KPA was not appropriate. In addition, it may be noted that during the life-cycle of a project, managers may find many opportunities to implement KPA goals. For example, practices related to planning


and tracking of plans may need to be followed more than once. Similarly, software configuration management and peer reviews KPA goals may need to be followed periodically. As a consequence, the need arises for a frequency scale to measure consistency in adopting the CMM practices. In this study, a frequency scale to measure the implementation of KPAs within a project is introduced. The anchors in the scale are defined based on the percentage of the time the KPA goals were implemented in a project. The specific anchor definitions are shown in Fig. 2. The measurement of the KPA goal questions on a frequency scale captures, to a large extent, the institutionalization of the KPA practices. Hence, there was no explicit need to include the KPA common features questions in this study. The rationale for excluding the common features questions is that they focus on institutionalization issues with categories such as commitment to perform, ability to perform, and so forth. While institutionalization is clearly important to overall process maturity, it focuses on the likelihood of consistent implementation in the future. In contrast, in this study we are interested in the consistency of actual implementation on each specific project, not in the likelihood of future implementation.3 The questionnaire used in this study is provided in a technical report [39]. In contrast to most process improvement studies that focus on process measurement at the organizational level, in this study the investigation is at the project level. Because some of the KPAs specified in the CMM (see Fig. 1), such as organization process focus, organization process definition, technology change management, and process change management, specifically address organization-wide process issues, we do not expect any variation on these KPAs within an organization. Thus, these KPAs are omitted in the empirical model reported here. The issue of validity and reliability of the process measures using the new scale is discussed in Section 4.2.

4.2

Validity and Reliability of Process Measurements One of the criticisms of process research is related to the reliability and validity of measurement [17]. Validity of a measurement is defined as the extent to which a measurement operation actually measures what it claims to measure. Reliability of a measurement operation expresses the degree to which a measure is consistent and repeatable. Although one may expect strong validity and reliability from using a popular process measurement framework such as the CMM, a formal test for validity and reliability of the CMM measures for process maturity is seldom addressed. This study fills this gap by addressing the validity and reliability of the process measures. The content validity of the CMM KPA goal questions is concerned with identifying to what extent the goal questions address the KPAs as perceived by the respondents. There are actually two questions related to the validity of CMM KPA measurements. The first question is as follows: 3. We would like to thank Dave Zubrow, Dennis Goldenson, Will Hayes, and Jim Herbsleb at SEI, and Barry Boehm at the Univeristy of Southern California, for their valuable input regarding our scale definition for process measurements.

805

Do the respondents understand the goal questions in the same context as defined in the KPAs?

For example, do the respondents perceive that goal questions for the requirements management KPA in the CMM actually address requirements management issues? The second question deals with the identity of the KPAs. Are the respondents able to distinguish each of these KPAs from one another?

In a recent study, factor analysis of KPA goal questions from the respondents of two organizations revealed that respondents were not able to separate software quality assurance, software configuration management and defect prevention [11]. In order to address these questions, a technique used in the information systems literature was adopted [15], [42]. This technique addresses the construct validity of KPA measures and also identifies ambiguous questions. Two pilot studies were conducted using this technique. In the first pilot study, four judges who had at least five years of industry experience in managing software projects were selected. The goal questions of the different KPAs were mixed together in a random way to form a single pool of questions. The judges were presented this pool of questions and a list of KPAs on a separate sheet of paper. Based on their own interpretation of the questions, each of the judges independently matched the questions to the KPAs. Prior to this exercise a standard set of instructions explaining the procedure was read to all the judges. This technique indicates to some degree the construct validity of KPAs in terms of convergence and divergence of items within the KPAs. That is, if a goal question was consistently placed within the same KPA by all the judges, then it was considered to demonstrate convergent validity within the respective KPA construct and discriminant validity with other KPA constructs. A method defined in Moore and Benbasat [42] was used to assess both the reliability of the goal question classification scheme and validity of the questions and KPA constructs. This method requires analysis of how many items were placed by the panel of judges correctly in the ªtargetº KPA. A higher percentage of items placed in the target construct indicates a higher degree of interjudge agreement across the panel. In addition, scales based on KPAs that exhibit higher ªcorrectº placement of items within them are considered to have a high degree of construct and content validity, with a high potential for good reliability scores. However, this procedure is more a qualitative analysis than a rigorous quantitative test for reliability and validity. There are no predefined guidelines for ªgood or sufficientº level of agreement, but the matrix of targeted vs. actual placement of the goal questions by the judges can be used to identify potential problem areas. As described earlier, we conducted the above goal question sorting exercise in two pilot studies. The matrix of sorted KPA vs. target KPA for all the questions by all the four judges in the first pilot study are shown in Table 1. In the table, the actual KPAs are listed in the first column and the target KPAs for the questions as placed by the judges are listed in the second row. Note that due to space limitations, only abbreviations of the KPAs are shown in the

806


VOL. 25,

NO. 6,


Fig. 2. Anchor definitions for frequency-based scale.

second row. For example, the third row in the table indicates that in the placement of the two requirements management KPA goal questions by four judges, the questions were placed correctly in five instances. These questions were twice placed within the software product engineering KPA and once in the software project planning KPA. The number of goal questions in each KPA is shown within parenthesis in the first column of Table 1. Ideally in a correct placement of all goal questions in the respective KPAs, the off diagonal cells of the square matrix (shown in bold) in Table 1 should be empty. However, it may be noted that certain questions are in the off-diagonal cells of the matrix, meaning that they were misplaced. The overall placement ratio at the end of this first pilot study was 75 percent. In order to avoid confusion and improve the clarity of the goal questions in the second pilot study, some additional information on the goals of the KPAs and

specific definitions of related items were provided to the judges. The judges in the second pilot study were software managers in the research site. Hence, this set of judges was a more close representation of people assessing KPA conformance in our study. The matrix of actual KPAs vs. target KPAs for all the questions by all the four judges in the second pilot study is shown in Table 2. The additional information on the KPA goals considerably improved the overall hit ratio this time to 84 percent. Hence, this additional information on KPA goals and definitions were included in the final process data collection. The hit ratios, the lowest of which was 75 percent, indicate an acceptable degree of validity. The reliability of the KPA constructs and goal questions measured on the frequency scale was evaluated by collecting more than two responses for each project in our study. A typical set of respondents for a project included a


807

TABLE 1 CMM KPA Goal Question Placements in Pilot Study 1

project manager or team leader and two software developers. The reliability among the multiple responses in the same project was checked using Cronbach's alpha coefficient. We found that in all the projects, the reliability coefficient was above the recommended threshold of 0.70 [44]. The reliability coefficients indicate the degree of agreement across different respondents in the same project.

5

DATA AND ANALYSIS

5.1 Data The research site in this study is one of the leading development firms in the software industry. This firm develops commercial software systems for various applications. A random sample of over 50 products was reduced to the final sample of 45 products due to incomplete data. The data collection period was 1993±1995. The software developed in all these projects were systems software and the programming language used was mostly C, with a few projects developed in a similar proprietary language. The data included scores on the frequency scale described in Section 4.1 (Fig. 2) for all the CMM KPA goal questions. In addition, the data included size of the products, field defects reported by the customers, and people factors such as personnel capability and experience. The details of these variables are provided next. The measure of product size (SIZE) is in thousands of nonblank, noncommented, source code lines. Although this measure has been widely used in the software engineering literature, certain problems are associated with this measure [36], [32]. The main shortcoming stems from the inaccurate and inconsistent definition of ªa line of codeº across various

programming languages and tools used to count the number of source lines of code. In this study, 80 percent of the projects were written in the C language and the rest were written in a proprietary language similar to C. In order to ensure consistency of measurement across products, the size of each product was measured using the same in-house size measurement tool. At the research site, the service support for software products is organized in two levels. Customer complaints are received at the first level and the problem is analyzed for its validity. Most often customer complaints may arise from misinterpretations of product features by the customers. Such invalid complaints are filtered out at the first level. Once a certain complaint is accepted as valid at the first level, the complaint is then passed on to second level service support. At this level, a detailed analysis of the reported problem is conducted under different environments. For example, the specific condition described by the customer is simulated in the test laboratory; the problem is then repeated. If, however, the reported problem is not repeatable in a laboratory environment, further information is sought from the customer to understand the conditions in which the problem occurs. On some occasions, even a data dump may be requested from the customer site to further analyze the problem. Once the problem is identified as a consequence of this analysis, the severity of the problem is assessed based on various parameters such as degree of disruption to the customer's business operation and product usage. The problems classified as defects in this level are included in our DEFECTS variable as the number of field defects. This variable included defects reported in the first 24 months after the product release. In addition,

808


VOL. 25,

NO. 6,


TABLE 2 CMM KPA Goal Question Placements in Pilot Study 2

since a single defect may be reported by more than one customer, such defects are counted only once. The personnel capability (PC) variable measures the technical capability of the members of the team. This relative construct for each team in the software development laboratory at the research site is measured using a five point Likert scale ranging from (1) very low capability to (5) very high capability. The middle score of (3) was defined as medium laboratory capability. A similar construct has also been used in earlier software cost models [4]. In each project, the product manager and at least two randomly selected software engineers rated the capability of the team. The final score for each team was obtained by averaging these three responses. The reliability index of 0.74 (Cronbach alpha) for this measure was well above the threshold recommended [44]. The process variable (PROCESS) is based upon the consistent adoption of practices specified in the CMM KPAs (see Fig. 1). Note that we have multiple responses on process conformance for each project. The measure for each KPA (for each project) is based on the average of responses (for that project) to the goal questions on the frequency scale shown in Fig. 2. The distribution of the CMM goal question responses for nine sample KPAs are shown in Fig. 3, Fig. 4, and Fig. 5. As per our expectations, we did not find much variation in responses to goal questions of some KPAs such as technology change management, organization process definition, and software subcontract management. Since, in this study, the consistency in adopting process improvement practices is addressed at a project level, the PROCESS variable measure is an average of the scores for a project on the following eight KPAs: 1) requirements management, 2) software quality

assurance, 3) software configuration management, 4) software project planning, 5) defect prevention, 6) training program, 7) peer reviews, and 8) software product engineering. The summary statistics including mean, median, minimum, maximum, and standard deviation statistics of these eight KPA scores based on the average responses to the goal questions in each project for these eight KPAs are presented in Fig. 6. The other 10 KPAs were omitted due to one or more of the following factors: .

the KPA was common across the organization and not expected to vary from project to project (e.g., organization process definition and technology change management KPAs in Fig. 5. . the KPA was widely viewed as not applicable in the firm's setting (e.g., software subcontract management in Fig. 5 . a substantial number of respondents selected ªdon't knowº for the KPA The distribution of the responses on the frequency scale also confirmed our belief that project managers may not be consistent in implementing KPA practices and a simple YES/ NO binary scale may not be appropriate. The summary statistics of the independent variables are shown in Table 3.

5.2 Model Estimation A linear specification of the model (1) implies that the effects of size and team factors on the number of defects are additively separable and linear. However, in the case of software development, increase in the size of the product may increase the product complexity substantially and hence, beyond a certain size, the effect on defects may be


809

Fig. 3. Distribution of responses to CMM KPA goal questions. The categories 1-5 are scales with anchors defined in Fig. 2. Catagory 6 is the combined ªdon't knowº and ªdoes not applyº responses.

nonlinear [43]. Similarly, beyond a certain level of team capability, the marginal effect of improving the capability further may not be the same. Moreover, for the data sample in this study, the normality test for error terms in a simple linear model rejected the model specification. Thus, the following generic multiplicative specification:

Through log transformation of the variables we get the following model specification.

DEFECTS e1 SIZE2 PROCESS3 PC4 e"1

The parameters of this model were estimated using ordinary least squares. Standard assumptions of these estimators were tested. Also, the presence of heteroskedas-

is adopted.

lnDEFECTS 1 2 lnSIZE 3 lnPROCESS 4 lnPC "1

2

810


VOL. 25,

NO. 6,


Fig. 4. Distribution of responses to CMM KPA goal questions. The categories 1-5 are scales with anchors defined in Fig 2. Category 6 is the combined ªdon't knowº and ªdoes not applyº response.

ticity was tested using White's [50] test and no evidence for the same was found. The presence of heteroskedasticity would indicate that the error terms and the exogenous variables are not independent. Multicollinearity is significant correlations among the independent variables. In our model, it may be argued that independent variables such as product size, personnel capability, and process may be correlated. For example, it may be argued that project teams with higher capability team members may follow a disciplined process. If there

was significant multicollinearity, then it would be difficult to correctly interpret the coefficients of (2). In particular, it would not be justifiable to claim that all three explanatory variables SIZE, PC, and PROCESS independently explain some of the variation in the number of defects. In addition, the presence of multicollinearity may pose other estimation problems that may lead to wrong signs of the parameter estimates. Hence, it is important to test for any significant effect of multicollinearity in our model. There are several methods to detect the effect of multicollinearity on model


811

Fig. 5. Distribution of responses to CMM KPA goal questions. The categories 1-5 are scales with anchors defined in Fig 2. Category 6 is the combined ªdon't knowº and ªdoes not applyº responses.

parameter estimates. We next describe the methods used in this study. The effect of multicollinearity in the above model was evaluated using two approaches specified in Belsley et al. [53]. In the first approach, the variance inflation factor (VIF) for each independent variable was computed. For each independent variable i, VIF is the ratio 1=1 ÿ R2i where R2i is the R2 from regressing the independent variable i on the other independent variables. A high value of VIF for any

independent variable indicates a significant correlation among the independent variables. Rules such as declaring muticollinearity as a problem if the highest VIF is greater than five have been proposed. In our model, the highest VIF was 1:09 indicating no evidence of multicollinearity. An alternate approach is to check for the condition number (the ratio of largest and smallest eigenvalue) of the XT X matrix where X is the matrix with the columns as independent variables in the regression model. Large values of condition

812


VOL. 25,

NO. 6,


Fig. 6. Summary statistics of CMM KPA scores at the project level N 45.

number indicates multicollinearity. Rules such as declaring significant multicollinearity if the condition number greater than 30 have been proposed [10]. In our analysis, the highest condition number is was well below 30 indicating no evidence for severe multicollinearity. The calculated values of F-statistics for the model exceeded the critical values at the 1 percent significance level. This indicates that our model explains a significant proportion of the variance in DEFECTS. The analysis of variance indicating the proportion of variance accounted for by the three explanatory variables is presented in Table 4. Note that size of the product explains a large proportion of variance in DEFECTS. This is also due to the fact that in our sample the SIZE variable takes on a larger range of values than the personnel and process variables that are measured on a five point ordinal scale. The parameter estimates of our model with the t-statistics are presented in Table 5. The positive sign of the SIZE coefficient indicates that larger TABLE 3 Summary Statistics

products exhibit higher number of defects. The negative signs of the PC and PROCESS coefficients indicate that the number of defects decreases with higher personnel capability and with higher process consistency with the CMM. We also computed the Cook's distance for each data point in our sample to check the influence of any outliers in our parameter estimates [10]. Cook's distance is computed by deleting each data point from the sample and reestimating the model to identify any significant difference in the parameter estimates. Analytically Cook's distance is the distance between new and original sets of parameters in the parameter space. The maximum Cook's distance in our sample was 0:13 indicating the absence of any single influential data point.

5.3 Model Results The findings in this field study provide evidence that the effect of personnel capability of the team and consistent adoption of the CMM practices in reducing the number of product defects may be significant. Our results indicate that larger products exhibit a higher number of defects. Although the number of defects are higher in larger products, the rate of increase in defects in larger products is not higher than the rate of increase in size (the absolute value of 2 is less than 1). Note that in our multiplicative specification, if 2 was equal to 1, that means an increase in defects is linear in size. Our findings also indicate that teams with relatively higher personnel capability are associated with products that exhibit a lower number of defects. The results indicate that software projects that consistently adopt the CMM practices exhibit significantly lower numbers of defects. Thus, our results provide a link between consistent software processes and reduced field defects in the resulting product. The relative comparisons of the effect of people, process, and size variables can be made only on the basis of standardized parameter estimates shown in the third row of Table 5. These estimates indicate


813

TABLE 4 Analysis of Variance

TABLE 5 Model Parameter Estimates

that size of the product has the largest influence whereas the effects of process and people variables on the number of defects are quite comparable. However, it should be noted that the units of measurements of various independent variables in our study are not the same; the cost of improving people and process variables may be significantly different. Hence, in order to assess the net business value from improving process or people, managers should consider both the costs and benefits of these improvements. There are important practical implications of this research for software managers and other practitioners. Our research results suggest that a project can improve the quality of its delivered product by more consistent implementation of the CMM practices (at least the eight KPAs ultimately included in our composite process variable) at a project level. Of the three exogenous variables, process consistency is generally more controllable by a project manager and by software developers than are size of the product or personnel capability of the staff. Thus, improving process consistency is a practical action that individual project managers and project teams can realistically take to enhance product quality. Our numerical results (Table 5) imply that holding size and personnel capability constant, a 1 percent improvement in process consistency score with the CMM goals would be associated

with a 0.95 percent reduction in number of defects. Note that since the model is in log transformed variables, the marginal effect from process improvement is not constant at all the points. The relationship between PROCESS and DEFECTS in our model indicates that holding other variables constant, a modest, and realistic, change from an average consistency rating of ªoccasionallyº to ªfrequentlyº would be associated with a significant reduction in the number of defects.

6

CONCLUSION

In this article, a discussion is presented on how inconsistent application of the CMM practices in a software process project can lead to both domain level and environment level process deviations. Further, various challenges in process consistency measurements and in measuring adoption of CMM practices at a project level are presented; a new frequency scale is introduced to account for prior limitations. Subsequently, an empirical model is presented that investigates the effect on product defects of consistent adoption of the CMM practices (measured on the new scale) in a project, product size, and personnel capability of the team. The results of this study, based on data from 45 projects in one of the leading software organizations,

814


indicate that projects that consistently adopt the practices specified in the CMM KPAs exhibit substantially lower numbers of defects. The study also indicates that, although larger products exhibit higher defects, the rate of increase in defects is lower for larger products. The results also indicate that project teams with higher personnel capability deliver products with lower number of defects. Thus, this paper provides evidence on how consistency of a software process may be improved and defects in the software may be lowered by regular adoption of the practices specified in the CMM KPAs. Some limitations of this research are as follows: 1) the sample size (45 projects) is modest, 2) the findings are based on data from a single firm, and 3) all the projects in the sample are from the systems software domain. In addition, the measures of personnel and process variables in our analysis are ordinal and subjective.4 Further research is required to define objective measures specifically through use of process support technology tools such as SETINEL and SPADE [14]. Specific features for collecting objective metrics on process inconsistencies need to be incorporated into process modeling tools and, subsequently, these objective measures should be used in empirical studies to link process inconsistency measures to project outcomes. In addition, it should be noted that the focus of the model presented in this paper is on the effect of process consistency on the number of defects in software products. However the number of defects captures only the reliability part of software qualityÐwhich is multidimensional [35]. Improvements in process consistency may also facilitate managers to enhance other dimensions of software quality such as functionality and usability. Other outcomes of a project such as productivity and cycle-time may also be significantly improved through process consistency and are issues for future research.

ACKNOWLEDGMENTS The authors are grateful to the managers and engineers at our research site for their support and valuable suggestions. The authors also thank the members of the software engineering measurement and analysis group and other colleagues in the Software Engineering Institute at Carnegie Mellon University for their useful suggestions in this study. Partial financial support for this study was provided through the University of Michigan Business School faculty grant, a fellowship from our research site, and support from the Department of Defense via the Software Engineering Institute at Carnegie Mellon University. Our research site also provided travel funds and significant management time for on-site data collection and further discussion of our results.

NO. 6,


REFERENCES [1]

[2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16]

[17] [18]

[19] [20] [21] [22] [23] [24] [25]

[26] 4. It may be argued that logarithmic transformation of ordinal variables is not appropriate. However, we estimated our model without the logarithmic transformations of these variables; our results still hold and there are no major changes to sign or magnitude of the parameter estimates.

VOL. 25,

[27]

S. Bandinelli, A. Fuggetta, C. Ghezzi, and L. Lavazza, ªSPADE: An Environment for Software Process Analysis, Design and Enactment,º Software Process Modeling and Technology, A. Finkelstein, J. Kramer, and B.A. Neuseibeh, eds., Taunton, U.K.: Research Studies Press, 1994. V.R. Basili, ªThe Experience Factory and Its Relationship to Other Quality Approaches,º Advances in Computers, M.V. Zelkowitz, ed., vol. 41, pp. 65-82, Boston: Academic Press, 1995. V.R. Basili, M. Zelkowitz, F. McGarry, J. Page, S. Waligora, and R. Pajerski, ªSEL's Software Process Improvement Program,º IEEE Software, vol. 12, no. 6, pp. 83-87, Nov. 1995. B.W. Boehm, Software Eng. Economics. Englewood Cliffs, N.J.: Prentice Hall, 1981. T.B. Bollinger and C. McGowen, ªA Critical Look at Software Capability Evaluations,º IEEE Software, pp. 25-41, July 1991. A.M. Christie, Software Process Automation: The Technology and Its Adoption. Berlin: Springer-Verlag, 1995. B.K. Clark, ªThe Effects of Software Process Maturity on Software Development Effort,º Univ. of Southern Calif., 1997. unpublished S. Conte, H. Dunsmore , and V. Shen, Software Engineering Metrics and Models. Menlo Park, Calif.: Benjamin/Cummings, 1986. J.E. Cook, L.G. Votta, and A.L. Wolf, ªCost-Effective Analysis of In-Place Software Process,º IEEE Trans. Software Eng., 1998. to appear R.D. Cook and S. Weisberg, Residuals and Influence in Regression. London: Chapman & Hall, 1982. B. Curtis, ªThe Factor Structure of the CMM and Other Latent Issues,º Proc. 1996 SEPG Conf., Atlantic City, N.J., 1996. B. Curtis, M. Kellner, and J. Over, ªProcess Modeling,º Comm. ACM, pp. 75-90, Sept. 1992. G. Cugola, E. Di Nitto, C. Ghezzi, and M. Mantione, ªHow to Deal with Deviations During Process Model Enactment,º Proc. 17th Int'l Conf. Software Eng., Seattle, New York: ACM, 1995. G. Cugola, E. Di Nitto, A. Fuggetta, and C. Ghezzi, ªA Framework for Formalizing Inconsistencies and Deviations in HumanCentered Systems,º ACM Trans. Software Eng. and Methodology, vol. 5, no. 3, pp. 191-230, July 1996. F.D. Davis, ªPerceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology,º MIS Quarterly, vol. 13, no. 3, pp. 319-340, Sept. 1989. K. El Emam and D.R. Goldenson, ªSome Initial Results from the International SPICE Trials,º Software Process Newsletter, Technical Council on Software Eng., vol. 6, IEEE Computer Society, Spring 1996. K. El Emam and N.H. Madhavji, ªReliability of Measuring Organizational Maturity,º Software ProcessÐImprovements and Practice, vol. 1, pp. 3-25, 1995. P. Ferguson, W.S. Humphrey, S. Khajenolori, S. Macke, and A. Matvya, ªIntroducing the Personal Software Process: Three Industry Case Studies,º Computer, vol. 30, no. 5, pp. 24-31, May 1997. A. Finkelstein, J. Kramer, and B. Nuseibeh, Software Process Modeling and Technology. Research Studies Press, U.K., 1994. C. Fox and W. Frakes, ªThe Quality Approach: Is it Delivering?º Comm. ACM, vol. 40, no. 6, pp. 25-29, 1997. R.L. Glass, ªThe Software Research Crisis,º IEEE Software, pp. 4247, Nov. 1994. D. Goldenson, ªA Multiple Response Scale for Process Measurement,º working report, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1994. A. Gopal, T. Mukhopadhyay, and M.S. Krishnan, ªThe Role of Software Process and Communication in Offshore Software Development,º Comm. ACM, 1998. J. Hatz et al., Trillium Method, Method Issue 1.0, Nortel BNR Corporate Standard 133.21 W. Hayes and J.W. Over, ªThe Personal Software Process (PSP): An Empirical Study of the Impact of PSP on Individual Engineers,º Technical Report, CMU/SEI-97-TR-001, Software Eng. Inst., Pittsburgh, 1997. T.G. Heineman, J.E. Botsford, G. Caldiera, G.E. Kaiser, M.I. Kellner, and N.H. Madhavji, ªEmerging Technologies that Support a Software Process Life Cycle,º IBM Systems J., vol. 33, no. 3, pp. 501-529, 1994. J. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes, and M. Paulk, ªSoftware Quality and the Capability Maturity Model,º Comm. ACM, vol. 40, no. 6, pp. 30-40, 1997.


[28] W.S. Humphrey, Managing the Software Process. Mass.: AddisonWesley, 1989. [29] W.S. Humphrey, A Discipline for Software Eng. Mass.: AddisonWesley, 1995. [30] Industry Surveys, ªSoftware Industry: A Huge Market, Getting Bigger All the Time,º pp. C108, Nov. [31] C. Inwood, ªStandards May Solve User Frustration,º Computing Canada, vol. 20, no, 2, 1994. [32] C. Jones, Applied Software Measurement: Assuring Productivity and Quality, New York: McGraw-Hill, 1991. [33] M. Keil, ªPulling the Plug: Software Project Management and the Problem of Project Escalation,º MIS Quarterly, vol. 19, no. 4, pp. 421-447, Dec. 1995. [34] M.I. Kellner, L.C. Briand , and J.W. Over, ªA Method for Designing, Defining, and Evolving Software Processes,º Proc. Fourth Int'l Conf. Software Process: Improvement and Practice, Brighton, England U.K., pp. 37-48, IEEE CS Press, Dec. 1996. [35] S. Kekre, M.S. Krishnan, and K. Srinivasan, ªDrivers of Customer Satisfaction for Software Products: Implications for Design and Service Support,º Management Science, vol. 41, no. 9, pp. 1,4561,470, 1995. [36] C.F. Kemerer, Software Project Management Readings and Cases. McGraw-Hill, 1997. [37] J. Kmenta, Elements of Econometrics. New York: Macmillan, 1986. [38] M.S. Krishnan, ªCost and Quality Considerations in Software Product Management,º Carnegie Mellon Univ., Pittsburgh, 1996. unpublished doctoral dissertation, [39] M.S. Krishnan and M.I. Kellner, ªSoftware Process Measurement: Implications for Improving Software Quality,º working paper, Univ. Michigan Business School, Ann Arbor, 1998. [40] M.S. Krishnan, ªThe Role of Team Factors in Software Cost and Quality: An Empirical Analysis,º Information Technology and People, vol. 11, no. 1, 1998. [41] P. Kuvaja, J. SimilaÈ L.L. Krzanik, A. Bicego, S. Saukkonen, and G. Koch, Software Process Assessment & Improvement: The Bootstrap Approach. U.K.: Blackwell, 1994. [42] G.C. Moore and I. Benbasat, ªDevelopment of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation,º Information Systems Research, vol. 2, no. 3, pp. 192-221, 1991. [43] A.M. Newfelder, Ensuring Software Reliability. New York: Marcel Dekker, Inc., 1993. [44] J.C. Nunnally, Psychometric Theory. McGraw-Hill, 1978. [45] G. Pall, Quality Process Management. Englewood Cliffs, N.J.: Prentice Hall, 1987. [46] M. Paulk, ªHow ISO 9001 Compares with the CMM,º IEEE Software, pp 74-83, Jan. 1995. [47] M. Paulk, B. Curtis, M.B. Chrissis, and C. Weber, ªCapability Maturity Model,º version 1.1, Technical Report, CMU/SEI-93-TR24, Software Eng. Inst., Pittsburgh, 1993. [48] D.E. Perry, N.A. Staudenmayer, and L.G. Votta, ªPeople, Organizations and Process Improvements,º IEEE Software, pp. 3645, July 1994. [49] D.M. Raffo and M.I. Kellner, ªModeling Software Processes Quantitatively and Evaluating the Performance of Process Alternatives,º Elements of Software Process Assessment and Improvement, ch. 16, pp. 297-341, K. El Emam, and H.N. Madhavji, eds., IEEE CS Press, 1999. [50] H. White, ªHeteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,º Econometrica, vol. 48, no. 5, pp. 817-838, 1980. [51] R. Whitney, E. Nowrocki, W. Hayes, and J. Siegel, ªInterim Profile Development and Trial of a Method to Rapidly Measure Software Engineering Maturity Status,º Technical Report CMU/SEI-TR-4, Software Eng. Inst., Pittsburgh, Mar. 1994. [52] D. Zubrow, W. Hayes, J. Siegel, and D. Goldenson, ªMaturity Questionnaire,º Technical Report CMU/SEI-94-SR-007, Software Eng. Inst., Pittsburgh, 1994. [53] D.A. Belsley, E. Kuh, and R. Ewelsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, 1980.

815

M.S. Krishnan received his MS and PhD degrees from the Graduate School of Industrial Administration, Carnegie Mellon University in 1993 and 1996, respectively. He is an assistant professor of computer and information systems at the University of Michigan Business School, Ann Arbor. His current areas of research interests include product development and maintenance issues in software products, design for quality, customer satisfaction measures in software, strategic issues, and business value of information. technology applications. Dr. Krishnan has published book chapters and several papers on software product development and stragic issues of Informaiton technology. His research publications have appeared in leading journals that include Management Science, Communications of the ACM, Harvard Business Review, and Decision Support Systems. He teaches a course on Network Infrastructure for e-Business in the MBA program, and Systems Analysis and Design in the BBA program at the University of Michigan. Marc I. Kellner holds a BS degree in physics, an BS degree in mathematics (both with university honors), an MS degree in computational physics, and an MS degree in systems sciences, all from Carnegie Mellon University. He received his PhD degree in industrial administrationÐSystems Sciences (specializing in MIS) from Carnegie Mellon University. He is a senior scientist at the Software Engineering Institute (SEI) of Carnegie Mellon University in Pittsburgh, Pennsylvania. He has pioneered and led much of the work on software process modeling and definition conducted at the SEI. Currently, he leads a team developing exemplary process guides for paper and for the Web. His research interests also include empirical studies of software process improvement, quantitative process model simulation, and software maintenance. He has published several book chapters and articles in the leading software journals, as well as numerous refereed conference proceedings papers. He has also delivered more than 100 technical presentations on software process issues at a variety of conferences world wide and has taught tutorials on process modeling, definition, and related topics to approximately 1,600 software professionals. He has served on the program committee of numerous conferences, workshops, symposia, and the like, and serves on the editorial boards of two journals: Software Process: Improvement and Practice and Empirical Software Engineering. Prior to joining the SEI in 1986, Dr. Kellner was a professor at Carnegie Mellon University, where he established and directed a BS degree program in information systems. He has also served on the faculty of the University of Texas, Austin, and consulted for several organizations.