Why software projects fail? Empirical evidence and relevant metrics ...

299

Proceedings of the IWSM - Mensura 2007

Why software projects fail? Empirical evidence and relevant metrics Maurizio Morisio, Evgenia Egorova, Marco Torchiano Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy {maurizio.morisio, eugenia.egorova, marco.torchiano}@polito.it

Abstract. In this paper we present an empirical study on factors that cause the success or failure of a software project. The study deals with the definition and validation of metrics and indicators to make more objective the meaning of success (and conversely failure) of a project, and correlates characteristics of a project (such as experience of the staff and the project manager, techniques used for requirement engineering, and so on) with its result. The study, performed on a number of Italian companies, can be easily replicated in other countries or within a company, in order to assess its software processes and steer its process improvement initiatives.

1

Introduction

Software metrics are meant to be measures in the scientific sense of the term, with the goal of giving objective views on software related objects and phenomena. The well known problem of software development is that there is very little scientific foundation to the subject. Electrical engineers can use physics as the foundation of their activities. As a result the measures they use in their daily work, such as Volt and Ampere, correspond to well defined phenomena (electric potential, electric current), and satisfy the rules and properties defined by theories of measurement, such as the representational theory [11]. The models they use, such as the Ohm’s law, are experimentally validated and expressed in mathematical form. In software engineering we have a hard time in defining the objects and properties (what is exactly the size of a software program? And the complexity?). Not surprisingly measuring an ill defined property is a hard task (are Lines of Code a valid measure of size?). Models relating properties are therefore hard to find and validate. Some models, for instance the relationship between effort and size, correspond to the common perception of engineers (larger programs require more effort) but are hard to express in a general form (is the relationship linear, exponential?). Other models are simply envisaged but still very fuzzy (what is the relationship between effort in integration testing and post release defect rate?). Initially, the software engineering community has attacked the problem of defining software measures as a hard science problem. The Software Science approach by Halstead [15] in the mid seventies could be mentioned as an example of the optimism in that period. However, these approaches were completely missing the validation part, and the importance of human factors.

300

Proceedings of the IWSM - Mensura 2007

The key point here is that software development is not a hard science, but it is a human, brain intensive activity. Human factors play a key role (it is well known that productivity varies of a factor of ten and more among developers [10]), therefore software development should be regarded more as a soft science, such as sociology, economics, and partially medicine. The misunderstanding here comes from several context factors: • Software runs on computers, hard objects based on hard science and hard technology. • A software program in itself is a hard object, always producing the same results with the same inputs (and we know this is not always true). • And finally software programs were developed, especially at the beginning of the profession, by engineers trained in hard sciences. But the process to develop a program is not a hard process, it is a design activity mostly based on human activity. On the other hand, calling software development a soft science may be a compliment to it, and a detriment to sociology and other soft sciences. Sociologists and economists collect and process a huge amount of hard data, that empirical software engineers simply dream of [7]. Recognizing these factors led some pioneer researchers (such as Vic Basili and Barry Boehm) to adapt experimentation to the software field. In practice the idea is to always start bottom up, observing software development in vivo (in industry) or in vitro (experiments with students) to identify the key factors that drive the process and possibly identify relationship among them. Seminal results include the Cocomo model [4], based on developing a model of effort vs. size using statistical analysis on data from a few dozens projects. And comparison of testing techniques [2]. In the ESE (Experimental Software Engineering) view, that simply uses the scientific empirical method, a metrics or a theory (or model) should be validated to be used successfully in industrial practice. In practice: 1. a metric (or a model) is proposed; 2. empirical studies are planned to validate it; 3. as evidence is built in favor, the metric or model can be used by other researchers or practitioners. The latter phase corresponds to general use of a metric/model in industrial practice, what traditionally is the object of interest by the industrial metric community. However, as for all scientific theories, a metric/model is considered valid temporarily, until a new experiment possibly falsifies it. One of the key questions for practitioners, and specifically for project managers, is what makes a project successful? Are there factors that can help to devise if a project will be successful? Research on success factors for software projects has been performed, and it is a notable example of ESE activities that can help practitioners. In this paper we present an example of an ESE study on success factors for software projects. The goal is twofold, present the current status of ESE research on this topic, and present and discuss the process to develop and perform an experiment. While this study has been performed on projects from different companies, it could be easily run by a software company on internal projects, as a way to start or assess a process improvement initiative.

301

Proceedings of the IWSM - Mensura 2007

2

State of the art

Forty years after the start of the discipline of software engineering, software projects still fail. And, according to Cusumano [9], they fail for reasons very similar now and at the beginning of the history of the discipline, usually situated at the Garmisch NATO software engineering conference in 1968. While a plethora of papers have been written about the topic, very often we can find a confusion between two quite different concepts: what we mean for a successful project or product (we will call this success indicator) and what causes a project to be successful (we will call these success or failure factors). Both indicators and success factors are, in the end, measures, and should be defined and validated accordingly. Despite the importance of analyzing key success and failure factors in software projects, there are a limited number of studies on the topic. The Chaos report [25] is the one with the largest sample of projects analyzed, and points out an impressive failure rate. However, the methodology followed by the study has never been published in detail. Glass [14] points out that the sample of projects considered could be biased towards failures. What is a project success? (or else, what are the indicators of a successful project?) The most adopted definition is the one of the Standish Group [25], i.e. within planned budget and schedule, and meeting business objectives. Kerzner [17] adds to the previous definition meeting performance requirements and obtaining user acceptance. So what is a project failure? Is a project that is “one day late, or one dollar over budget, or one requirement short” [20] an unsuccessful one? As Ryan Nelson [20] states, definition of success or failure is a very subjective factor, and it depends on the stakeholders. Different authors have asked software developers, managers and other stakeholders about their perception of success. Several studies have been made to analyze successful/unsuccessful projects and to define key success factors. Agarwal and Rathod [1] define three groups of stakeholders: Programmers/Developers, Project Managers and Customer Account Managers. The authors conclude that for these stakeholders “meeting the scope of software project, which comprises the functionality and quality of the project outcome, is the highest determinant of success”. Their study is based on 105 answered questionnaires from 19 organizations. Procaccino et al. [23, 21, 22] have conducted in 1999, 2001 and 2003 three empirical studies and have analyzed responses of 21, 31 and 76 IT professionals for their perception of success. The later studies have confirmed the results of the earlier ones. The findings suggest that the most important factors are mainly two: “namely the importance of personal aspects of the work and customer/user involvement” [21]. Personal aspects include, for example, the feeling of doing a good job and professional growth. Besides, there are some differences between the managers and developers perception of project’s success. From the developers’ point of view the most important factors are realistic expectations of customers and users, and well defined scope of the project. Managers find most important participation of customers and users in estimating project schedules [23]. User involvement is not only indicator of the project success but also an important critical factor. Besides this one, critical

302

Proceedings of the IWSM - Mensura 2007

factors include sponsor support, project manager authority, enough time for requirement elicitation [23], clear and understandable requirements, skilled team, feedback from project manager and defined methodology [24]. Verner et al. [28] analyzed 102 projects in USA and Australia. They find as success factors a well defined scope for the project, confidence of the customer in the development team, correct management of the requirements, and effective risk management by the project manager. Verner and Evanco [27] surveyed 101 respondents about 122 projects. They found the following factors explaining the major part of projects’ success: good project manager and his communication skills, good requirements and project vision. Berntsson-Svensson and Aurum [3] have performed an analysis on 27 projects in 15 Swedish and Australian companies. They find that key success factors are complete and precise requirements, good size estimation, and involvement of the customer. They also use a more comprehensive definition of success. Beyond the standard, ‘internal’ definition of successful project (within schedule, within budget), they add an ‘external’ view: adding value for the customer, measured in term of customer satisfaction. Taylor [26] analyzed 25 outsourced IT projects in Hong Kong. They identify as success factors schedule and budget management, vendor staffing, understanding of requirements, client expectations, and change management. From a literature review Nah et al. [19] define top management support, business plan and vision, effective communications as critical software project success factors. Wohlin and Andrews [29] have added factors such as project planning and ability to follow the plans, as crucial for the project success. Besides, there are other studies, like NASA report [18] with DOs and DON’Ts for the project success. In addition to what other authors have pointed out, they suggest mainly minimizing the bureaucracy and not relaxing the quality goals. The last one demotivates developers, since most of them are quality oriented. In summary we have seen that the standard definition from project management literature of the project success, such as “in time, within budget”, was extended by many researchers to include also other stakeholders points of view. For developers, a project is successful when they understand requirements, are not under schedule pressure, do a good job and grow professionally. For the managers, user satisfaction is an indicator of a project’s success. Robert Glass has proposed this formula to summarize all: “User satisfaction = quality product + meets requirements + delivered when needed + appropriate cost.” [12]. Critical factors that explain projects’ success are mainly: customer trust and involvement; experienced managers that perform risk, schedule and requirements management and negotiation; complete and precise requirements; good estimation of effort and workload; support from sponsors and top management.

3

The case study

Our goal is to gain more knowledge on the issue of failures and successes of software projects. On one hand the previous studies have a limited scope, both in

303

Proceedings of the IWSM - Mensura 2007

number of projects considered and in their geographical location. On the other hand software development has changed in the last years, because of outsourcing, offshoring, emphasis on web based, distributed applications, integration of open source or commercial components. Besides, from a methodological point of view the results of the previous studies can only partially be merged. The model underlining all studies is the one reported in Figure 1, but factors and projects success or failure are defined in different ways.

factor

Software project result = success or failure

Figure 1. The model relating factors and result of software project. Therefore, the starting point of any study could be a careful definition of success or failure, and a comprehensive definition of factors. As discussed in the previous section, the more general definition of the result of a project considers two points of view z internal, or strictly related to project management: satisfaction of budget, scope and schedule constraints z external, or more related to customer satisfaction: the project produces a software product or service that is used by and is useful for the customer Unfortunately there are too many different definitions of success (both internal and external) therefore in our study we do not provide a definition of success. We prefer to ask the respondents to point out whether a project is successful or not. In other words here the goal is to develop a definition of success from the answers of the interviewees, instead of imposing our definition. Next, a study has to provide a list of factors that can influence the result of a project towards success or failure. We use the same list of factors of [3] in order to be able to combine the results form the two studies. The factors are listed in Table 1 together with a short description. Basically these factors cover the size of the project (in term of effort spent and number of staff involved), the project manager (in terms of his/her experience), the requirement engineering process, and the project management process (including if the project ended up on time and within budget). In clear, the research question underlying this study is: “What is the effect of certain factors on the success or failure of projects?”

304

Proceedings of the IWSM - Mensura 2007

Metric

Type

Description Dependent S_F Nominal {FAILURE, Outcome of the project: SUCCESS} Independent PM_EXP Ordinal: 6-points Likert scale Experience of PM PM_INSIGHT Does PM understand client/market Ordinal: 6-points Likert scale requests PM_CHANGE Boolean Was PM changed PM_EXTRA Boolean Did PM allow extra hours STAFF_EXTRA Boolean Were extra hours paid REQ_DEF Tools/methods used for requirements Boolean elicitation and definition REQ_INIT Requirements collected before Boolean starting project REQ_AFTER If not before, requirements completed Boolean during project REQ_RES_TIME Enough time devoted to requirements Boolean definition OBJ_DEF Boolean Objectives well defined PROJ_PLAN Boolean Project plan was well defined PLAN_QUAL Ordinal: 6-points scale Project plan quality IN_BUDGET Boolean Project ended within budget IN_TIME Boolean Project ended within schedule STAFF_ADD Staff added in-course to keep up with Boolean schedule CUST_PAY Boolean Customer paid for development CUST_INVOLV Customer was closely involved in the Boolean project RISK_IDENT Risks were identified at project Boolean beginning

Table 1: Success factors. In other words, are there any actions, behaviors, attitudes that are clearly different in a successful project than in a failed one? To support this comparison of successful and failed projects, each respondent was requested to consider two projects he knew well, one failed and one successful; and answer twice to the questions, one per project. We designed the empirical investigation as a case control study: as already stated each interviewee answers about a successful project and a failed one. With such a design we will not be able to find out the percentage of projects that fail. That figure is of course considered strictly confidential by software organization and it is outside the scope of our study. The questions are framed in a questionnaire, largely inspired by [3], that is made of 4 sections z General information about respondent and company z Information about a successful project

305

Proceedings of the IWSM - Mensura 2007

z z

Information about a failed project Features that a successful project or product should possess (in general)

The questionnaire was administered, over the period January 2007 to March 2007, using telephone and personal interviews. The contacts selected were software related companies located in north-west of Italy and sampled randomly from the Italian yellow pages database. Actually we collected many more measurements than presented here; for the sake of space we focus only on metrics that are relevant to the research question under consideration. In particular (as shown in Table 1) we have one dependent variable (i.e. success of failure of the project) and a set of independent variables representing mostly success factors. Actually two of them (whether the project was completed within time or budget) are success indicators. 3.1

Results

We received answers from 14 companies about 38 projects (20 successful and 18 failed). The staff size per project ranged from 2 to 20 with an average of 5.5 persons. The effort ranged from 2 to 24 person-months with an average of 10. The majority of projects were bespoke (25), some were mixed (6) or in-house (5), and just a few market-driven (2). This distribution well reflects the Italian software industry we know from our direct experience. We performed statistical tests to identify which, if any, of the independent variables could be a discriminant in determining success of failure of a project. For variables measured on a 6-point scale we applied the Mann-Whitney test, for boolean variables we built contingency tables and applied the Fisher exact test. In all the tests we fixed an alpha level of 5%. The analysis was performed using the R-statistical package1. The results from the tests are summarized in Table 2. The statistically significant differences are highlighted in bold and with colored background. An important difference with the original work presented in [3] originates from the structure of the questionnaire: we sampled one failed and one successful project per respondent. This approach enabled us to meet the condition to apply statistical tests to identify the discriminating factors. The results from our analysis confirmed several laws and hypotheses presented in the software engineering literature [10] and previously hinted or validated by other studies cited in section 2.

1

http://www.r-project.org/

306

Proceedings of the IWSM - Mensura 2007

Median Success Failure Test M-W 5 4 M-W 5 4 Only 1 in 38 projects PM_CHANGE changed PM In all projects PMs allowed PM_EXTRA extra hours STAFF_EXTRA Fisher REQ_DEF Fisher REQ_INIT Fisher REQ_AFTER Fisher REQ_RES_TIME Fisher OBJ_DEF Fisher PROJ_PLAN Fisher PLAN_QUAL M-W 5 3 IN_BUDGET Fisher IN_TIME Fisher STAFF_ADD Fisher CUST_PAY Fisher CUST_INVOLV M-W 5 4 RISK_IDENT M-W 4 3 Metric PM_EXP PM_INSIGHT

p-value 0.28 0.12 0.65 0.47 0.05 0.01 0.47 0.16