Quantitative models for early prediction of software ...

Quantitative Models For Early Prediction of Software Development Success: A Practitioner’s Perspective

Dissertation Submitted to the Faculty of Drexel University by J. Drew Procaccino in partial fulfillment of the requirements for the degree of Doctor of Philosophy October 2002

© Copyright 2002 J. Drew Procaccino. All Rights Reserved.

ii

Acknowledgements

I thank Dr. Marvin Darter (Rider University) who, in addition to serving on my committee, has also been instrumental in my collegiate career from both sides of the podium. Thanks also to Dr. June Verner for her support and guidance throughout the program, including her invaluable service as my Committee Chairperson. Thanks also to Committee members Dr. Michael Atwood (Drexel), Dr. Scott Robertson (Drexel), and Dr. Lauren Eder (Rider University), for their direction and support. I consider a few other faculty members of Drexel and Rider as ‘honorary’ members of my Committee because of their guidance and support during my research, and they include Dr. Willian Evanco (Drexel), Dr. Katherine Shelfer (Drexel), Dr. David Gefen (Drexel) and Dr. Donald Wise (Rider). Also, my thanks to Dr. Katherine McCain (Drexel) and the staff of the Dean’s office for their administrative expertise, the College of IST’s Computer Resource Center and Sloan Center for their expertise in all things technical, and to my colleagues in the doctoral program, including Jodi Williams, Brian Finnegan, Lisa Morton, Yolanda Jones and Steve Brusstar. While writing my dissertation, I moved from an adjunct to full-time faculty position in The Department of Computer Information Systems at Rider University (Lawrenceville, NJ), and I wish to thank the Department for its consideration and guidance while I conducted my research, and in particular, Dr. James Dailey and Dr. William Amadio.

A special thanks to my family, including my uncle and aunt, Basil and Ann Ferrara, who have been like grandparents to me, and my aunts, Margaret Campbell and Mary Alice Karnas, for their many years of support. And lastly, to my mom, Jean Ann Procaccino, and my late father, John Alan Procaccino, for their love, support and guidance, which included getting me started on this educational path.

iii

Table of Contents

LIST OF TABLES......................................................................................................................... vii LIST OF FIGURES ........................................................................................................................ ix ABSTRACT.................................................................................................................................... xi 1.

PROBLEM STATEMENT .................................................................................................... 1

2.

DEFINING PROJECT SUCCESS......................................................................................... 5 2.1 Project Stakeholders....................................................................................................... 8 2.2 Process vs. Product ...................................................................................................... 13 2.3 Technical vs. Non-Technical Issues............................................................................. 14 2.3.1 Project Managers and Training .......................................................................... 15 2.3.2 The High-Tech Illusion ...................................................................................... 16 2.3.3 Managing Technology vs. Managing People..................................................... 17 2.3.4 Difficult Nature of Developing Software........................................................... 17

3.

NON-TECHNICAL COMPONENTS OF SOFTWARE DEVELOPMENT ...................... 19 3.1 Sponsor/Management Support and Participation......................................................... 23 3.1.1 Sponsor/Management Components.................................................................... 23 3.2 Customer/User Support and Participation.................................................................... 32 3.2.1 Customer/User Components .............................................................................. 33 3.3 Requirements Management.......................................................................................... 42 3.3.1 Requirements Management Components........................................................... 44

4.

MAIN STUDY: OVERVIEW ............................................................................................. 54 4.1 Project Focus................................................................................................................ 54 4.1.1 Project Stakeholders........................................................................................... 55 4.1.2 Process vs. Product............................................................................................. 56

iv

4.2 Defining Project Success ............................................................................................. 56 4.2.1 Pilot Study #1: Software Practitioner’s Success Factors.................................... 57 4.3 Importance of Study..................................................................................................... 60 4.4 Anticipated Outcomes.................................................................................................. 62 5.

MAIN STUDY: RESEARCH MODELS, QUESTIONS AND HYPOTHESES ................ 64 5.1 Theoretical Research Model ........................................................................................ 64 5.2 Empirical Research Model........................................................................................... 65 5.3 Research Questions ...................................................................................................... 66 5.4 Support For Research Questions.................................................................................. 69 5.5 Hypotheses................................................................................................................... 73

6.

MAIN STUDY: DATA COLLECTION INSTRUMENT ................................................... 75 6.1 Industry Contacts ......................................................................................................... 75 6.2 Distribution Mechanism............................................................................................... 76 6.3 Survey Instrument........................................................................................................ 77 6.3.1 Respondent Demographic Variables .................................................................. 78 6.3.2 General Project Variables................................................................................... 79

7.

MAIN STUDY: DATA ANALYSIS................................................................................... 84 7.1 Research Questions and Independent/Dependent Variables........................................ 84 7.2 Validity and Reliability................................................................................................ 86 7.3 Processing of The Raw Data........................................................................................ 88 7.4 Respondent and Organizational Demographics ........................................................... 90

8.

MAIN STUDY: ANALYSIS AND FINDINGS.................................................................. 93 8.1 Overview of Analytical Methods................................................................................. 93 8.1.1 Cronbach’s Alpha/Factor Analysis .................................................................... 93 8.1.2 Bivariate Correlation Analysis ........................................................................... 94

v

8.1.3 Ordinal Regression Analysis.............................................................................. 95 8.1.4 Bayesian Belief Networks.................................................................................. 96 8.2 Results of Analytical Methods..................................................................................... 98 8.2.1 Cronbach’s Alpha/Factor Analysis .................................................................... 98 8.2.2 Bivariate Correlation Analysis ......................................................................... 106 8.2.3 Ordinal Regression Analysis............................................................................ 113 8.2.4 Bayesian Belief Network ................................................................................. 126 9.

CONCLUSIONS................................................................................................................ 145

10.

RECOMMENDATIONS FOR FURTHER STUDY ......................................................... 152

LIST OF REFERENCES............................................................................................................. 154 APPENDIX A: Various Perspectives on Defining Software Development Project Success ...... 163 APPENDIX B: Summary Results of The Standish Group’s CHAOS Study [1995a]................. 165 APPENDIX C: Results From Standish Group’s Unfinished Voyages [1995b].......................... 166 APPENDIX D: Pilot Study #1: Software Practitioner’s Success Factors – Survey Instrument.............................................................................................. 167 APPENDIX E: Pilot Study #1: Software Practitioner’s Success Factors - Descriptive Stats..... 172 APPENDIX F: Interviews With Software Practitioners ............................................................. 190 APPENDIX G: Proposed Causal Model From Procaccino, et al [2002a]................................... 195 APPENDIX H: Proposed Causal Model From Procaccino, et al [2002b]................................... 196 APPENDIX I: Pilot Study #2: Other Project Variables - Survey Instrument............................ 198 APPENDIX J: Pilot Study #2: Other Project Variables - Descriptive Statistics........................ 204 APPENDIX K: Data Dictionary of National Database (from Applied Computer Research) ..... 224 APPENDIX L: Main Study: Survey Instrument......................................................................... 226 APPENDIX M: Main Study: Data Dictionary of Survey Variables ............................................ 231 APPENDIX N: Summary of Recoded/Edited Variables............................................................. 233 APPENDIX O: Main Study: Descriptive Statistics..................................................................... 236

vi

APPENDIX P: Frequencies of Survey Responses...................................................................... 237 APPENDIX Q: Frequencies Used To Populate Bayesian Belief Network ................................. 257 APPENDIX R: Probabilities Propagated Through Proposed Bayesian BeliefNetwork ............. 372 APPENDIX S: Ordinal Regression Analysis of Critical Path From Bayesian Belief Network ............................................................................................................. 374 APPENDIX T: Overall Impact On Project Success.................................................................... 375 VITA

........................................................................................................................................ 377

vii

List of Tables

1. Top 10 Highest Ranked Success Variables ............................................................................. 58 2. Success-Related Project Variables With Lowest Coefficient of Variation ............................. 59 3. Variables From Survey Section 1............................................................................................ 78 4. Research Questions and Independent/Dependent Variables ................................................... 84 5. Summary of Organizations and Respondents Providing Usable Responses ........................... 89 6. Automatic Recording of Irregular Responses ......................................................................... 90 7. Total Variance Explained (Sponsor/Management) ............................................................... 100 8. Rotated Component Matrix (Sponsor/Management) ............................................................ 100 9. Total Variance Explained (Customer/User) .......................................................................... 101 10. Rotated Component Matrix (Customer/User) ....................................................................... 101 11. Total Variance Explained (Requirements Management) ...................................................... 103 12. Rotated Component Matrix (Requirements Management).................................................... 103 13. Total Variance Explained (Survey Sections 1, 2 and 3)........................................................ 104 14. Rotated Component Matrix (Survey Sections 1, 2 and 3)..................................................... 105 15. Bivariate Correlations (Spearman’s) For Research Question 1............................................. 107 16. Bivariate Correlations (Spearman’s) For Research Question 2............................................. 107 17. Bivariate Correlations (Spearman’s) For Research Question 3............................................. 108 18. Bivariate Correlations (Spearman’s) For Research Question 4............................................. 109 19. Bivariate Correlations (Spearman’s) For Research Question 5............................................. 109 20. Bivariate Correlations (Spearman’s) For Research Question 6............................................. 110 21. Bivariate Correlations (Spearman’s) For Research Question 7............................................. 110 22. Bivariate Correlations (Spearman’s) For Research Question 8............................................. 111 23. Bivariate Correlation (Spearman’s) For Practitioners’ Success Variables............................ 112

viii

24. Bivariate Correlation (Spearman’s) For Practitioners’ and Organizational Success Variables .................................................................................................................. 113 25. Causal Relationships In Proposed Model.............................................................................. 114 26. Ordinal Regression Results For Research Question 1 (dependent variable Q4.01) .............. 116 27. Ordinal Regression Results For Research Question 2 (dependent variable Q4.02) .............. 117 28. Ordinal Regression Results For Research Question 3 (dependent variable Q4.03) .............. 118 29. Ordinal Regression Results For Research Question 4 (dependent variable Q4.08) .............. 119 30. Ordinal Regression Results For Research Question 5 (dependent variable Q5.02) .............. 120 31. Ordinal Regression Results For Research Question 6 (dependent variable Q5.03) .............. 121 32. Ordinal Regression Results For Research Question 7 (dependent variable Q5.05) .............. 122 33. Ordinal Regression Results For Research Question 8 (dependent variable Q6.01) .............. 123 34. Overall Ordinal Regression Results ...................................................................................... 125 35. Summary of Ordinal Regression Models .............................................................................. 126 36. Frequency of Independent Variables (Q3.02, Q3.03 and Q5.01).......................................... 128 37. Frequency Distribution For Q6.01, “Practitioners’ Overall Perception of Success” ............ 130 38. Original Probability Propagation........................................................................................... 131 39. Impact Summary For Q6.01, “Practitioners’ Overall Perception of Success” ...................... 134 40. Impact Summary For Q5.02, “Agreement on Requirements Reached” ................................ 136 41. Impact Summary For Q4.02, “Level of Customer/User Particpation Was High”................. 137 42. Impact Summary For Q4.08, “Customer/Users Made Adequate Time” ............................... 138 43. Summary of Ordinal Regression Models On Critical Path ................................................... 140 44. Summary of Variables’ Impact On Project Success.............................................................. 141 45. Summary of Project Components’ Impact On Project Success............................................. 142

ix

List of Figures

1. Framework For Defining Project Success ................................................................................. 6 2. Interrelations of Major Project Stakeholders............................................................................. 9 3. Conceptual Components of Bayesian Belief Networks .......................................................... 63 4. Theoretical Research Model.................................................................................................... 65 5. Empirical Research Model ...................................................................................................... 66 6. Proposed Correlations For Research Question 1..................................................................... 67 7. Proposed Correlations For Research Question 2..................................................................... 67 8. Proposed Correlations For Research Question 3..................................................................... 67 9. Proposed Correlations For Research Question 4..................................................................... 67 10. Proposed Correlations For Research Question 5..................................................................... 68 11. Proposed Correlations For Research Question 6..................................................................... 68 12. Proposed Correlations For Research Question 7..................................................................... 68 13. Proposed Correlations For Research Question 8..................................................................... 69 14. Sample Conditional Probability Table From Hugin System ................................................... 97 15. Proposed Bayesian Belief Network With Bivariate Correlation Coefficients....................... 129 16. Overall Propagated Probabilities Through Hugin’s Monitor Windows................................ 132 17. Percentage of Q6.01 When All Cases of Each Independent Variable=4 or 5 ....................... 135 18. Percentage of Q5.02 When All Cases of Each Independent Variable=4 or 5 ....................... 136 19. Percentage of Q4.02 When All Cases of Each Independent Variable=4 or 5 ....................... 137 20. Proposed Bayesian Belief Network With Critical Path and Non-Significant Paths.............. 139 21. Percentage of Q6.01 When All Cases of Each Project Component=4 or 5........................... 143

x

Abstract Quantitative Models For Early Prediction of Software Development Success: A Practitioner’s Perspective J. Drew Procaccino June M. Verner

This study investigated some of the early non-technical components of the software development process from the perspective of project success. The causal relationships among these components, as well as their relationship to software practitioners’ overall perception of project success, were depicted through a Bayesian Belief Network. Components related to sponsor/management, customer/users and requirements management. Data was gathered through a survey of practitioners. This research is important because it provides graphic and quantitative findings of the ‘downstream’ implications of the investigated components on the probability of project success. The study’s working definition of success reflects elements of software development that practitioners have indicated are important to them and, by extension, motivate them. Furthermore, motivation has been shown to have the single greatest impact on staff productivity, which has direct implications for the ability of organization to deliver software products on time, within budget and that mets customer/user requirements.

The software industry continues to be plagued by projects that are characterized by cost overruns, excessive time to complete (if they get completed at all), poor reliability/quality and/or failure to meet agreed upon business objectives/requirements. The cause of most project failures has little to do with technological issues, despite the tendency among project managers to focus on the technical issues involved in software development. Further, there is a lack of quantitative research into the early, on-going and non-technical components of software development projects, specifically from the perspective of software practitioners.

xi

The proposed model provided quantitative evidence that agreement on requirements being reached between customer/users, a high level of customer/user participation, and users that make adequate time for requirements gathering had the three largest direct impacts on practitioners’ overall perception of project success. The chain of project components that had the largest impact on project success was having a sponsor throughout the project, users that make adequate time for requirements gathering, a high level of customer/user participation in the development process, and agreement on requirements between customer/users and the development team.

1

1

1. Problem Statement

The software industry continues to be plagued by projects that are characterized by cost overruns, excessive time to complete (if they get completed at all), poor reliability/quality and/or failure to meet agreed upon business objectives/requirements [Barki, et al 1993; Jones 1995; Standish Group 1995a; McConnell 1996; Glass 1998; Verner, et al 1999; Jiang and Klein 2000; Ravichandran and Rai 2000]. Poor project outcomes continue despite numerous studies that document and dissect failed projects [Lyytinen 1988; Barki, et al 1993; Standish Group 1995a; Brooks 1995; Jones 1995; Nidumolu 1996; Amoako-Gyampah, et al 1997; Jiang, et al 2001]. Successful software development projects are usually defined as having met the agreed upon business objectives/requirements, and completed on time and within budget [Keider 1974; Pinto and Slevin 1988; Standish Group 1995a; Jones 1995; Clavadetscher 1998; Baccarini 1999; Linberg 1999; Leffingwell and Widrig 2000; Wohlin, et al 2000].

Even though there is a tendency among project managers to focus on the technical issues involved in software development, including those related to hardware and software (such as programming language and compilers) [DeMarco and Lister 1999], the cause of most project failures has little to do with technological issues [DeMarco 1991; McConnell 1996; Warne and Hart 1996; Linberg 1999; Sumner 1999; DeMarco and Lister 1999]. Although research suggests that failed projects suffer from the poor management of people-related problems rather than technical problems [DeMarco 1991; McConnell 1996; Linberg 1999; Sumner 1999; DeMarco and Lister 1999], it is important to analyze project failures from all perspectives. Poor software development practices places organizational resources, such as time, money and the pool of software practitioners (programmers, database developers, system analysts, etc.) at risk. Practitioners become burned out, de-motivated and are likely to have decreased personal

2

productivity. This may lead to increased staff turnover, which again leads to lower (team) productivity. The end-result is that time, money and organizational goodwill are placed at further risk. Unfortunately, commonly used productivity and cost metrics only provide project managers with a ‘snap shot’ of the current state of the project. These metrics do not provide managers with information related to project components that would help to explain, “how the software has evolved into [its present] state” [Curtis 1980]. These metrics also do not explain anything about the motivation or lack of it among the members of the development team.

There is a general lack of quantitative research into the early, on-going and non-technical components of software development projects, specifically from the perspective of software practitioners. Amongst others, the relationships between some of a development project’s major non-technical risks, and the corresponding quantitative evaluation of the potential of projects to meet cost, time and functionality objectives, have not been widely examined in the IS literature [Jiang and Klein 2000]. Furthermore, quantifying risk is an important first step in assessing risk [Barki, et al 1993]. This study is intended to determine if quantitative findings can support the construction of a causal model of software development that would support (or refute) widely published anecdotal evidence regarding early non-technical components that contribute to project success. The quantitative findings of this study are based on information gathered through surveying software practitioners regarding recently completed development projects in which they were professionally involved. Categories of components related to some of the early aspects of the development process include sponsor/management support and participation, customer/user support and participation, and requirements analysis. Survey responses are measured on a fivepoint Likert scale, ranging from 5 (‘agree’) to 1 (‘disagree’).

Literature review and interviews with software practitioners provide support for the development

3

of a causal model. The form of the model is a probability-based Bayesian Belief Network, which is based on mathematical rules of calculating conditional probabilities, given at least some amount of empirical evidence. Graphically, the model is composed of several causal ‘sub’relationships. Essentially, the frequencies of the responses to each of the questions within the survey instrument, taking into account the various independent/dependent relationships among those variables, are used to populate the proposed Bayesian model. Each project component included within the model is represented as a survey question. Through the ability to generate and quickly update complicated quantitative relationships on modern desktop computers, the completed model can easily perform ‘what-if’ analysis in order to isolate the probable effect of a component (variable) on overall project success from the perspective of software practitioners (given the collected empirical data), which is this model’s ultimate dependent variable).

The ‘what-if’ analysis mentioned above facilitates investigations to determine (1) the ‘critical path’ (of chain of events) of proposed components that lead to project success, (2) the general category of investigated software development (sponsor/management support and participation, customer/user support and participation or requirements analysis) that has the largest collective impact (both direct and indirect) on project success, and (3) the single investigated component of software development that has the largest impact on project success. These findings are important because they can assist managers in early evaluation of on-going projects and enable them to address the investigated development issues. ‘What-if’ analysis, as well as the proposed Bayesian model itself, can help graphically and quantitatively demonstrate the potential ‘downstream’ implications of the actions (or inactions) of management on the development process, particularly in regard to important non-technical aspects. This study is also intended to enlighten project managers in regard to the importance of practitioners’ overall perception of project success and practitioner motivation. This is an important relationship to both the developing organization and

4

the customer because motivation has been shown to be the single greatest contributor to staff productivity [Boehm 1981; McConnell 1996]. Moreover, productivity has direct implications for the cost (of which practitioners represent the single largest source) and time to develop software.

5

2. Defining Project Success The complex nature of finding a definition of success due to the need to consider product stakeholders is important when looking at software development projects. This discussion is organized as follows. Section 2.1: Project Stakeholders is presented first, followed by a comprehensive definition of project success, including two different major perspectives, (1) the process/product perspective (Section 2.2) and, (2) the technical/non-technical perspective (Section 2.3).

A concisely focused definition of project success within the context of software development is required in order to conduct this study. However, it is first appropriate to present a discussion of some considerations and components of project success. Specifically, there are three major factors that influence the notion of project ‘success’ (see Figure 1) and any analysis intended to lead to a working definition of project success must consider the following: •

The perspective of one or more project stakeholders, influenced by the culture, practices and system-related goals of the organization being asked to define success [Klein and Jiang 2001],

•

The development process and/or the resulting software product,

•

The Non-technical and/or technical focus.

The difficulty of measuring (and defining) all relevant success metrics (including those related to strategic advantage, advances in knowledge and process improvements [Klein and Jiang 2001]) and the project’s particular stage in the lifecycle [Pinto and Mantel 1990] add complexity to the factors in the above list. A simplified version of these factors and their interactions are depicted in Figure 1. Stakeholder perspective influences his/her perception of project success, as all aspects of project development are filtered through this perception. As a result, “Various stakeholders” is shown as the outermost rectangle in Figure 1. The next rectangle includes the particular definition and metrics that are used to define success. Process and product-related factors of the

6

development process can be considered in relation to a particular stage of the product lifecycle. Lastly, after considering the stage of the lifecycle, the non-technical and/or technical factors of development can be considered.

Figure 1: Framework For Defining Project Success

Projects that meet agreed upon business objectives, and are completed on time and within budget make up the generally accepted industry ‘standard’ organizational/managerial definition of project success [Keider 1974; Pinto and Slevin 1988; Standish Group 1995a; Jones 1995; Clavadetscher 1998; Baccarini 1999; Linberg 1999; Leffingwell and Widrig 2000; Wohlin, et al 2000]. This definition has traditionally been used because it is, in part, relatively easy to measure [Pinto and Slevin 1988]. User satisfaction is the single most widely cited measure of system success in the information systems literature [Jiang, et al 2001].

An additional success consideration for many software development managers of is that a project does not result in cancellation. In the strict sense of intention to design, construct and deliver a completed software-based product, termination should be considered a failure, at least to some extent. Regardless of whether a project is considered to be a failure or not, cancellation is clearly

7

not the most desirable outcome for any project stakeholder. Although not directly related to this study, it is interesting to note that there is some debate regarding the relationship between project termination and project failure. Although the Standish Group’s [1995a] widely cited research equates project cancellation with failure, there may be many useful lessons to be learned from an abandoned project [Ewusi-Mensah 1997]. Boehm [2000] suggests that canceling a project should not necessarily equate to project failure, as many projects fail due to changing conditions and “assumptions” [Pinto and Mantel 1990]. Several important software development factors may be, at least partially, beyond the control of a project manager, including the level of user participation, the lack of organizational resources and changing requirements (particularly in a “climate of rapid change”). Boehm [2000] also suggests that the notion of cancellation can be viewed as caused by bad management, and this encourages managers of failing projects to continue developing the project in order to not be perceived as a bad manager. Such perceptions can damage the manager’s career. However cancellation, may be a sign of good management as the cancellation of a project as early as possible may conserve organizational resources that are at risk due to an “infeasible project” [Boehm 2000]. Further, canceling the project may also allow for these resources to be utilized more effectively elsewhere. (In general, cancelled or late projects tend to be associated with larger systems, where a ‘large’ project refers to 5,000 functions points or approximately 500,000 source lines of procedural code. [Jones 1995]). However, in the American culture, there can be an emotional stigma associated with risky projects, as “owning up to risks is often confused with defeatism” [Boehm and DeMarco 1997]. In the forward for Bob Glass’, ComputingFailure.com [2001], Tom DeMarco said: “Our cultures guide us to think only of success, to concentrate on winning, not losing. The Plan For Success mentality sounds great, but it makes risk management almost impossible. And risk management is your most effective tool in a risk-intensive world. To do real risk management, you have to develop a deep understanding of the factors that have undone those who have gone before you, understand how these factors acted and what measures proved insufficient to contain them. If such factors proved fatal to your predecessors, they may prove equally fatal to you.”

8

Such a cultural climate may cause software development managers to be reluctant to admit that their project is facing some degree of risk. As a result, it is hardly surprising that risk may not be effectively managed, even though many researchers have described the potential negative impacts of inadequate assessment of project risk on the software development process [Ginzberg 1981; Boehm 1991; McConnell 1996; Keil, et al 2000a]. Too often software managers have had their development staff grind away at a project in the face of an unrealistic schedule [Pressman 1996; Glass 2001a]. This ‘risk-challenged’ mentality is often combined with overly optimistic practitioners, who believe they can accomplish more in a given period of time than is possible [McConnell 1996]. Clearly, many, if not all, of the project stakeholders are unlikely to look upon such projects as a success.

2.1

Project Stakeholders

The variety of project stakeholders adds additional parameters and associated complexity to the task of defining project success. Figure 2 is based on a schema for system evaluation developed by Klein and Jiang [2001], and it includes the following typical project stakeholders: Senior/Executive Management:

Oversee project managers, provide political support for this project, and may interact with MIS management, project managers and/or customer/users.

Project Manager:

Oversee the project development team and interacts with customer/users.

Customer/user:

Pays for and/or uses the completed software system.

Software Practitioner:

Includes programmers, database designers and system analysts.

Referring to Figure 2, Senior Management interacts with the Software Project Manager, who interact with both Developers/IT Professionals and the Customer and Users. Developers and IT Professionals interact with their Software Project Manager and the Customer and Users. Finally, the Customer and Users interact with the Software Project Manager and Developers and IT

9

Professionals. In some instances, there is also interaction between Senior Management and the Customer and Users.

Figure 2: Interrelations of Major Project Stakeholders (dotted lines indicates path not always present)

Each of these groups brings different backgrounds, and has different expectations and understanding of any metric that might be collected in order to evaluate success [Ginzberg 1981; Klein and Jiang 2001; Jiang, et al 2001]. Management is largely concerned with keeping their higher-level supervisor(s) happy (may include senior/executive management). In the case of a system being developed for an outside customer, management is concerned with keeping the people paying for the project (customer) happy. If the system is being developed for an in-house ‘customer’, then management needs to keep the management of the intended end-users happy. Software practitioners, however, bring their own somewhat unique perspective to the notion of success.

Each of these stakeholders plays a role in shaping practitioner’s typical perception of the relative success of a given software development project. While research suggests that practitioners value

10

the meeting of agreed upon business objectives [Linberg 1999; Procaccino and Verner 2002], studies have demonstrated that practitioners do not necessarily judge the relative success of a software project by the common project success definition of delivering a product on schedule and, within budget [McConnell 1996; Linberg 1999; Procaccino and Verner 2001a]. A practitioner’s perception of project success, in contrast to that of their management, tends to take a more micro-level project view, as practitioners are on the ‘front line’ of design and production. Practitioners may find considerable value and success in a project that is delivered late, overbudget and does not necessarily meet agreed upon business objectives [McConnell 1996; Linberg 1999]. Most practitioners have a general need to create things, particularly things that other people will find useful [Brooks 1995], as the nature of development work is creative [Pressman 1998]. It has been suggested that, “…programming…is fun because it gratifies creative longings built deep within us and delights sensibilities.” [Brooks 1995]. Practitioners also share in the “joy of always learning” [Brooks 1995]. Further, they enjoy working in a such a “tractable medium [as software]…only slightly removed from pure thought-stuff” [Brooks 1995]. The Myers-Briggs Test Indicator (MBTI), which is an often-used test to assess different psychological types [Lyons 1985], describes computer professionals as having a much higher preference for thinking than feeling as compared to general population [Boehm 1981]. Practitioners are likely to place higher value in the intrinsic value of the work itself rather than in extrinsic factors, which include compensation, working conditions and appropriate technical resources [McConnell 1996].

Linberg [1999] has provided evidence of a gap between the software practitioner 's definition of project success and the generally accepted industry definition presented earlier. He used development projects reports, interviews and surveys of eight development team members from a single organization to assess what contributes, from the perspective of practitioners, to a

11

successful project. Participants were asked, “What was the most successful project that you have worked on, and why?”; the common themes were: •

The project was a technical challenge.

•

The product worked the way it was intended to work.

•

The team was small and high performing.

Research suggests the following, from the perspective of practitioners, are additional contributors to successful projects: •

Effective leaders [McConnell 1996; Linberg 1999].

•

“Conductive organizational behavior” [Linberg 1999 (cites Rasch and Tosi 1992; McLean, et al 1996)].

•

Requirements that are technologically realistic [Linberg 1999].

•

Realistic estimation of scheduling and effort [Linberg 1999 (cites Boehm 1981; Glass 1992; Brooks 1995); Pressman 1996; Glass 2001a].

•

Necessary resources were available, including sufficient software personnel [Linberg 1999 (cites DeMarco and Lister 1999)].

•

Diverse and synergistic development teams [Linberg 1999 (cites Hohmann, 1997)].

Participants in Linberg’s study [1999] were also asked, “What was the least successful project that you have worked on, and why?”, and the common themes were as follows: •

Poor [project] management.

•

Poor marketing research [relates to developing systems that do not match the customer/users technical platform; i.e. hardware and/or operating system].

Linberg developed a project success continuum from the practitioner’s perspective, which includes whether the project was completed or cancelled. For each of these two outcomes, projects were rated as one of the following: •

Failed: completed projects that were characterized by the development of a product that “causes customer discontent” due to lack of perceived quality.

•

Low success: cancelled projects that were characterized by practitioners that did not learn anything new which could be applied to a subsequent project.

•

Successful: completed projects that were characterized by average cost, effort and schedule performance when compared to the industry as a whole.

12

•

High success: cancelled projects that were characterized by new knowledge for practitioners and artifacts that could be applied to future development work.

•

Exceptionally successful: completed projects that were characterized by “meeting all quality, cost, effort and schedule expectations”. According Linberg’s analysis, cancelled projects could not be classified as exceptionally successful.

Other research has noted that from a project success perspective, practitioners value several personal-related components of software development (including sense of achievement, perception that good, quality work was done) [McConnell 1996] and customer/user-related components (including customer/users have realistic expectations, a project meets all customer/user requirements and customer/users are involved) [Procaccino and Verner 2002]. Further, these components of success can be further categorized as being related to the process of developing software and its associated management, the development team, the individual practitioner (personal and professional needs) and the completed product. The following provides a more detailed list of the major components that contribute to the practitioner perception of project success. •

Management (both senior/executive and project management) The process-related components of project development important to practitioner’s perception of project success include quality of supervision [Herzberg 1987; Couger 1988; Ferratt, et al 1999], effectiveness of project management [Agarwal and Ferratt 1998],], participation and support of senior/executive management [Barki, et al 1993; Jiang, et al 1996; Ewusi-Mensah 1997].

•

The development team, including their peer relationships [Procaccino and Verner 2002] and the skill-level of the team [Anderson and Narshimhan 1979; McFarlan 1981; Davis 1982; Boehm 1989; Barki, et al 1993; Jiang, et al 1996; DeMarco and Lister 1999].

•

Personal and professional needs, including motivators [Linberg 1999; Procaccino and Verner 2002].

•

The completed product, including its reliability and user satisfaction [Bailey and Pearson 1983; Baroudi and Orlikowski 1988; Igbaria and Baroudi 1995; Jiang, et al 2001]. Much of the research that focuses on the practitioner’s perception of project success explores, to some extent, employee motivation. In general, the practitioner’s perception of project success is, at least in part, determined by components that are related to their motivation, and motivation has

13

the single largest impact on practitioner productivity [Boehm 1981; McConnell 1996]. “Motivation is the means by which the potent wellsprings of human energy and creativity are directed toward people’s desired goals” [Boehm 1981]. Herzberg [1987] describes motivation as being “…based on growth needs. Motivation is an internal engine, and its benefits show up over a long period of time. Because the ultimate reward [of] motivation is personal growth, people don’t need to be rewarded incrementally [such as through raises and promotions].” As an internal growth need, motivation stands in contrast to a ‘surface’ “fear of punishment or failure to get extrinsic rewards” [Herzberg 1987].

Motivational components include the nature of the work itself [Boehm 1981; McConnell 1996], having a personal sense of achievement [Boehm 1981; McConnell 1996; Glass 1999; Procaccino and Verner 2002], having a sense of doing a good job (i.e., delivering quality [Glass 1999; Procaccino and Verner 2002; Procaccino and Verner 2001b], and having the acceptance by the development team that the requirements are realistic and achievable [Pressman 1996; Linberg 1999; Procaccino and Verner 2002].

2.2

Process vs. Product

Baccarini [1999] and Wateridge [1998] suggest that software development and ultimate project success can be generally framed in terms of process and product. A complete evaluation of system success requires multiple measures [Pressman 1996; Linberg 1999; Klein and Jiang 2001; Jiang, et al 2001], and this is a complicated undertaking. As a result, separate, comprehensive studies are required to adequately investigate the concept of project success. There are many examples from software engineering research that define or make use of a wide variety of definitions of software project success. These definitions can generally be grouped by their

14

process or product-related perspectives (see below). Appendix A provides a sampling of this research. Process/project management:

Product:

including meeting cost, schedule and quality objectives, quality of project management process itself and the satisfaction of project stakeholders needs as they relate to project management process [Boehm 1981; Baccarini 1999]. including effects of the final product as they relate to meeting the goals and purpose of the system [Boehm 1981; Baccarini 1999], business outcomes (including impacts at both the organizational and individual levels), technical performance of the system, efficiency of the product’s operations (considering cost, time and productivity), end-user satisfaction with the completed system and the personal satisfaction of development staff (including professional growth, and challenging and interesting work) [Jiang and Klein 2000].

Of course, the components of process and product are intertwined, as the final product is the ultimate result of the development process itself.

2.3

Technical vs. Non-Technical Issues

The technical issues of software development include those directly related to hardware and software. Non-technical issues relate to people and managerial-related components of the development process. Non-technical, people-related components of the software development process tend to be under-managed. And several reasons for this include: 1. Project managers often lack managerial training, particularly in the realm of software development. 2. The “High Tech” illusion, which DeMarco and Lister [1999] suggest is prevalent in the information technology field. 3. Managing technical issues tends to be more straightforward than managing people. 4. Software development is a difficult, conceptual undertaking. [Brooks 1995]. 2.3.1

Project Managers and Training

Project managers are generally not trained to manage the job, but rather are trained, and have experience, in how the job is done [DeMarco and Lister 1999]. Historically, many organizations

15

have rewarded employees (often technical people) with opportunity to manage, though not all of these people are ‘management material’ [Ridings and Eder 1998]. It has been suggested that the “criteria, climate and rewards” associated with being a high performing software engineer are often not compatible with good managerial characteristics [Kellner 1991]. As a result, the information technology field has promoted many practitioners with little or no managerial training or experience to managerial positions. (The Peter Principle, which suggests that in a hierarchy, every employee tends to rise to their level of incompetence [Peter and Hull 1969], represents a common violation of the Principle of Job Matching. The latter Principle suggests that management should be aware of the need to create a work environment, including task assignment, that match the “skills and motivation of the people (software practitioners) available” [Boehm 1981]). The Principle of Job Matching also has ramifications for practitioner’s perception of project success. The end-result, as Boehm suggests, is “poor management can increase software [development] costs more rapidly than any other factor” [Boehm 1981], as its effects ripple throughout the development process, including scheduling, estimating, and team management and motivation [Kellner 1991]. In short, management skills (or lack of) have direct implications for project risk management, and ultimately project success. Boehm suggests that successful projects are often cited as having good risk managers [Boehm 1991].

The Standish Group [www.standishgroup.com] continues to conduct studies of domestically developed software projects and they note that most projects fail due to a lack of “skilled project management”, rather than money and/or technological tools/personnel. Skilled management includes the selection and utilization of an appropriate development methodology [Glass 1994; Ewusi-Mensah 1997]. This further speaks to the need for knowledgeable managers and effective managerial practices, as schedule and budgets are often padded in order to merely appear successful (from an organizational/managerial perspective) and well managed at the end of the

16

project [Brady and DeMarco 1994]. When a project is acknowledged to be late, it is generally not due to development effort by practitioners, but rather more often to a lack of a sound, well thought out scheduling effort [Brady and DeMarco 1994]. As a result, care should be taken when evaluating the relative success of a project from an organizational/managerial perspective.

Mismanagement can include pressure-packed schedule estimates [McConnell 1996; Glass 2001a] and/or the application of an inappropriate, perhaps overly restrictive, development lifecycle methodology [DeMarco and Lister 1999]. Such practices often leave practitioners de-motivated [McConnell 1996] and unhappy, which can lead to burnout, the need to train new staff, staff turnover [McConnell 1996] and late delivery [Keider 1974]. Because software practitioners usually construct software in teams [Lyons 1985] and group dynamics are important to efficient project development [Ewasi-Mensah 1997; DeMarco and Lister 1999], turnover interferes with the formation of cohesive and productive development teams, which in turn hampers productivity [DeMarco and Lister 1999].

2.3.2

The High-Tech Illusion

DeMarco and Lister [1999] described the “High-Tech Illusion” as a second reason why managerial effort and emphasis is so often placed on technical issues. That is, anyone who is professionally involved in a relatively new technology, such as software development, believes that he or she is in an “intrinsically high-tech business” [DeMarco and Lister 1999]. As a result, they tend to over-manage technical issues and lose sight of the critical, and on-going role that people, particularly software practitioners, play in these ‘high-tech’ businesses.

2.3.3

Managing Technology vs. Managing People

17

A third reason for under-management of the non-technical issues is that managing technical issues tends to be more straightforward than managing people, who ‘come pre-packaged’ with their unique personalities, strengths, weaknesses and opinions [DeMarco and Lister 1999]. Related to this are the managers of development projects who have some difficulty in relating to practitioners, in terms of their different professional roles within the development process. In addition, the practitioners’ perception of project success does not necessarily match that of project manage or the more senior management within the organization for which the system is being developed [McConnell 1996; Glass 1999]. Managers who attempt to motivate their development team as they themselves would prefer to be motivated are not likely to succeed [McConnell 1996].

2.3.4

Difficult Nature of Developing Software

A final reason for problems with software development project and the over-management of technical issues and under-management of non-technical, people-related issues is the problem that software development is, by its very nature, a difficult, conceptual undertaking. Part of the difficult nature of software development relates to what Brooks calls its “essence”, which refers to the difficulty that is inherent in the very nature of developing software [Brooks 1995]. The “essence” of software development relates to “the mental crafting” of software, which is “a construct of interlocking concepts: data sets, relationships among data items, algorithms and invocations of functions”. Brooks compares programming software to writing poetry, work that is “…only slightly removed from pure thought-stuff”. (Other challenges of developing software relate to the process of implementing the completed product, which is ‘external’ to the mental work of developing software. Brooks refers to such external challenges as “accidental” [Brooks 1995].)

18

The next chapter discusses some of the important non-technical considerations that have an early and on-going influence on the software development process.

19

3. Non-Technical Components of Software Development Previous software engineering studies have suggested a number of early, non-technical factors that contribute to the eventual success or failure of information systems development projects [McConnell 1996; Pressman 1997; Glass 1998; Verner, et al 1999]. The non-technical factors can be broadly categorized as follows: •

Sponsor/management support and participation (people and process-related),

•

Customer/user support and participation (people and process-related),

•

Requirements management (people and process-related),

•

Project manager and relationships with development staff (people and processrelated),

•

Estimation and schedule (people and process-related),

•

Software development process (process-related), and

•

Software development personnel (people-related).

The importance of sponsor/management support and participation, customer/user support and participation, and requirements management of successful software development is reflected in studies by the Standish Group [1995a and 1995b], as well as work done by Verner, et al [1999], Procaccino and Verner [2001a] and Procaccino, et al [2002a and 2002b; see Appendix G and H]. Many of the project components within three categories, namely (1) sponsor/management support and participation, (2) customer/user support and participation, and (3) requirements management of successful software development, are under the control of management. However, unsuccessful projects tend to deal inadequately with one or more of these issues. Unsuccessful projects tend to repeat the problems that Brooks outlined in 1975, including overly optimistic estimates (resulting in lack of calendar time), and inadequate project planning and change management [Brooks 1995].

The Standish Group’s widely cited CHAOS Report [1995a] illustrates the importance of management support, customer/user participation and requirements management through their

20

survey of a broad group of 365 information technology executive managers. These managers represented 8,380 development applications. (Several managers participated in focus groups and a few case studies were also included in the report.) Each survey respondent was asked to list reasons why their projects were ‘successful’ (i.e. delivered on time, within budget and met initially specified requirements), why they were ‘challenged’ (i.e. in danger of being delivered late and over-budget, and not meeting customer/user requirements) and why they were ‘impaired’ (i.e. cancelled). The most commonly cited reasons for ‘successful’ projects included user involvement, executive management support and clear statement of requirements. The most commonly cited reasons for ‘challenged’ projects included lack of user input, incomplete requirements and specifications, and changing requirements and specifications. Finally, the most commonly cited reasons for projects to be classified as ‘impaired’ were incomplete requirements, lack of user involvement and lack of resources. See Appendix B for further details.

The Standish Group’s follow-up to their CHAOS Report [Standish Group 1995a], Unfinished Voyages [1995b], revisited the importance of management support, customer/user participation and requirements management for successful project completion. IT executive managers were asked to list five ways to achieve each of the top ten-rated success criteria that had been determined in the original CHAOS Report. The results were then used to form a list of questions that a project manager should attempt to answer in order to arrive at a composite success score for a particular project. The categories noted above, namely user involvement, executive management support and clear statement of requirements are the first three included on the CHAOS questionnaire. See Appendix C for details.

Procaccino and Verner [2001a] and Procaccino, et al [2002a and 2002b; see Appendix G and H] also investigated the importance of sponsor/management, customer/users and requirements

21

management within the software development process, from the perspective of software practitioners. Procaccino and Verner [2001a] investigated some early development risks. In their analysis of 109 completed software development projects from various organizations across several industries. Respondents were asked to evaluate whether they considered the project to be successful. (However, respondents were not provided with, nor asked to provide, a specific definition of success.) Categories of their survey questions included Management Support, Customer/Users and Requirements.

Procaccino and Verner [2001a] used variables that exhibited a significant correlation with practitioners’ perception of success to generate a logistic regression equation, which correctly predicted success in 86% of the cases and explained 48% of the variance of project success (R2 = 0.48). The variables in the equation including the following: •

The level of confidence the customer has in the development team.

•

The level of involvement of customer/users in the development process.

•

The size of the project negatively affected requirements elicitation.

Procaccino, et al [2002a; see Appendix G] included a subset of the dataset used in Procaccino and Verner [2001a]. This research investigated 42 development projects from a single financial institution. Variables that exhibited a significant correlation with practitioners’ perception of success were used to generate a logistic regression equation, which correctly predicted success in 85% of the cases and explained 48% of the variance in success (R2 = 0.49). The variables in the equation included the following: •

The customer/users had realistic expectations.

•

If requirements were not complete and accurate, were they completed adequately?

•

The sponsor commitment lasted throughout the project.

22

It is interesting to note that although different variables were used in the logistic regression analysis, the results from Procaccino and Verner [2001a] and Procaccino, et al [2002a; see Appendix G] were very similar. Both studies demonstrated that a relatively small number of variables could predict a relatively high percentage of successful projects (85%).

Procaccino, et al [2002b] also examined the impact of sponsor/management support and participation, customer/user support and participation, and requirements management on practitioners’ perception of success. The data used in this study included that used in Procaccino and Verner [2001a] with an additional eleven respondents for a total of 120 from a number of different organizations. (Logistic regression analysis was not included in this study.)

The studies described above illustrate the importance of non-technical components in software development, particularly those related to the early impact of sponsor/management, customer/users and requirements management; people are the central theme throughout these three categories. This ties in with Boehm [1991], who suggests that, in general, “good people, with good skills and good judgment, are what make projects work”. McConnell [1996] also suggests that people are the most important element in successful development projects, as practitioners create, modify and apply the software while they interact with their management and customer/users of the system. However, people also represent the largest single cost in software development [Brady and DeMarco 1994]. Hence, it is important that management understand what is important in motivating practitioners in order to support a productive and creative work environment [McConnell 1996]; a productive and creative support environment in turn can lower the overall risk associated with a successful software development. All of this research points to the need for project managers to have some understanding of the development process from the perspective of software practitioners.

23

The remainder of this section discusses the importance of sponsor/management support and participation, customer/user support and participation and requirements management in the software development process. As noted earlier, neither Procaccino and Verner [2001a] nor Procaccino, et al [2002a; see Appendix G], as well as other cited studies, explicitly define project success from the perspective of software practitioners. However, Procaccino and Verner [2001b; 2002] and personal interviews [Interviews 2001] provide evidence of specific components of the development process that contribute to practitioner’s general perception of project success or failure. These components include: a sense of achievement, a sense of doing good job, a project with a plan, a well-planned project and a development team that accepted the requirements as being realistic/achievable. Therefore, when there is evidence in the literature that a particular component of the development process contributes to the relative success or failure of the project, we can infer a casual relationship between that component (as an independent factor) and project success the five ‘success factors’ from Procaccino and Verner [2002b; 2002].

3.1

Sponsor/Management Support and Participation

Sponsor/management support and participation extends beyond any single development project, as there is an on-going organizational need to acquire, allocate and prioritize resources in order to improve the development process and meet organizational objectives [Humphreys 1990].

3.1.1

Sponsor/Management Components

This section discusses the software development components related to Sponsor/Management Support and Participation reviewed for this study. The project had an upper-level (management) sponsor/champion:

24

A sponsor/champion is also related to a practitioner’s perception of the importance of other stakeholders’ participation in the decision-making process [Interviews 2001]. Research suggests that this component is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Developers believed they did a good job [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed on time [McConnell 1996; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Sponsor/champion stayed throughout this project.

•

There was a project plan [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project was well planned [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

High level of customer/users participation during development [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Customers/users made adequate time for requirements gathering [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project met requirements of customer/users [Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

There was an upper-level (management) sponsor/champion throughout this project. Sponsorship that does not last right through a project may contribute to delaying the project’s completion [McConnell 1996]. Further, practitioners have reported the importance of a sponsor/champion from their process perspective [Procaccino and Verner 2001a; Interviews 2001; Procaccino, et al 2002a; Procaccino, et al 2002b]. This is particularly true of strong sponsorship (Rainer and Watson 1995; Interviews 2001]. Having sponsor/champion throughout a project also relates to the importance practitioners place on having other stakeholders participate

25

in the decision-making process [Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•

Project completed on time [McConnell 1996; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•


•


•

High level of customer/users participation during development [Procaccino, et al 2002a].

•

A high confidence level in the project manager/team members was held by the customer/users [Procaccino, et al 2002b].

•

Participating customers/users stayed throughout the project [Procaccino, et al 2002a].

•

Customers/users made adequate time for requirements gathering [Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•


The upper-level (management) sponsor/champion was committed to this project. A committed sponsor or champion is important because he or she impacts a project both early, and right through, the development process [Rainer and Watson 1995; McConnell 1996; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b]. Sponsor participation can support more realistic scheduling and resource planning by helping to stop highranking managers from forcing the project manager/development team to accept unrealistic schedule changes or other such undermining changes [McConnell 1996; McKeen and Guirmares

26

1997]. For a similar reason, sponsor participation can also help with adequate change control practices and the beneficial introduction of new development methods [McConnell 1996].

Sponsor commitment can be reflected by sponsor participation in the decision-making process, which can in turn contribute to user buy-in and better resource planning [McKeen and Guimares 1997]. A committed sponsor can help to encourage commitment and participation from other project stakeholders [Procaccino and Verner 2001a; Interviews 2001; Procaccino, et al 2002a; Procaccino, et al 2002b]. As noted earlier, the benefits associated with sponsorship are more likely to occur with strong sponsorship [Rainer and Watson 1995; Interviews 2001]. Practitioners have also reported the importance of a sponsor/champion [Procaccino and Verner 2001a; Interviews 2001; Procaccino, et al 2002a; Procaccino, et al 2002b]. This component also relates to practitioner’s perception of the importance of having other stakeholders participation in the decision-making process [Lyytinen 1988; Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•


•


•

High level of customer/users participation during development [Procaccino and Verner 2001; Interviews 2001; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•

Requirements were accepted by the development team as realistic/achievable [Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

The project had a project manager:

27

Research suggests that the presence of a project manager is important for successfully meeting the various success criteria for a successful project [Keider 1974; McConnell 1996; Verner, et al 1999; Interviews 2001]. Lack of a project manager can lead to failure to plan for, and obtain, the required necessary human and technical resources. In addition, schedule slippages will not be identified and necessary adjustments will not be made [McConnell 1996]. However, the mere presence of a manager is not enough, as misguided management actions, such as adding additional development personnel to a project that is already behind schedule, can increase the cost of developing software very quickly [Boehm 1981]. In addition, a project manager must cultivate an atmosphere of communication between the development team and customer/users, which will facilitate effective requirements management [Pressman 1998]. There is also a need for negotiation skills in working with stakeholders on scheduling and requirements, budget and all other project activities [Pressman 1998]. Project managers also have the important responsibility of defining specific roles for members of the development team [Ewusi-Mensah 1997; Jiang and Klein 2000], and tracking progress through their support for on-going status meetings and status reports [McConnell 1996]. Practitioners have reported the importance of this component from their perspective of the development process [Verner, et al 1999; Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Verner, et al 1999; Interviews 2001].

•

Developers believed they did a good job [Verner, et al 1999; Interviews 2001].

•

Project completed on time [Keider 1974].

•

Project completed within budget [Keider 1974].

•

There was a project plan [Verner, et al 1999; Interviews 2001].

•

Project was well planned [Verner, et al 1999; Interviews 2001].

•

Adequate communication between project manager/team and customer/users [Pressman 1998].

•

Requirements were accepted by team as realistic/achievable [Verner, et al 1999; Interviews 2001].

28

•

Project met requirements of customer/users [Keider 1974].

There was a project manager(s) throughout this project (not necessarily the same person). As noted earlier, research suggests that the presence of a project manager is important for successfully meeting the various success criteria of a given project [Keider 1974; McConnell 1996; Verner, et al 1999; Interviews 2001]. Similar to the impact of sponsor/champion, the impact of management support and participation, both at the senior/executive [Standish Group 1995a] and project levels, is important because its impact is important because its impact is felt both early, and right throughout, the development process [Rainer and Watson 1995; EwusiMensah 1997; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b]. Similar to the impact of sponsor/champion(s), manager support and participation that does not last throughout the project may contribute to delaying the project’s completion [McConnell 1996]. Practitioners have reported the importance of this component from their perspective of the development process [Verner, et al 1999; Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•

Project completed on time [Keider 1974].

•

Project completed within budget [Keider 1974].

•


•


•

Adequate communication between project manager/team and customer/users [Pressman 1998].

•

Project met requirements of customer/users [Keider 1974; Verner, et al 1999; Interviews 2001].

The project manager(s) was knowledgeable/experienced overall in the application area. A knowledgeable and experienced project manager is important to meeting various project objectives [Ewusi-Mensah 1997]. This component is important because a good project manager

29

can anticipate what might go wrong prior to its happening [Pressman 1997; Glass 1998]. This implies knowledge of systems development and/or knowledge of the specific application area [Verner, et al 1999]. An experienced project manager can also assist in planning, including estimating resources, time and effort and cost [Pressman 1997]. Hohmann [1999] addresses another component of experienced project managers, which is the knowledge they can, and should, impart to rookies. Although not investigated in this study, Moynihan [2002] addresses numerous constructs of project risks through semi-structured interviews with experienced project managers. This research included the following (dependent) variables: •

Client’s understanding and clarity of their problem.

•

Commitment of project sponsor.

•

IT experience of customer/users.

•

Controlling party of the development project (practitioners, customer, other).

•

Practitioner knowledge of the application.

This component is important from the developers perception of project success [Verner, et al 1999]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Verner, et al 1999].

•

Developers believed they did a good job [Verner, et al 1999].

•

There was a project plan [Verner, et al 1999].

•

Schedule estimates were [un]realistic [Pressman 1997].

•

Requirements were accepted by team as realistic/achievable [Verner, et al 1999].

Overall, the project manager(s) supported the development team. In order for the development team to work as effectively and efficiently as possible towards organizational goals, it is important that project management support the team [Ginzberg 1981; DeMarco and Lister 1999; Jiang and Klein 2000]. Project managers need to establish a vision for the development team, anticipate that this vision may need to be revised in the future, hold the

30

team responsible for results (not the individual, as teams tend to set higher standards than individuals), delegate tasks to the team in a manner that are “challenging, clear and supportive”, don’t micro-manage individual practitioners tasks and remove barriers to team productivity when necessary [McConnell 1996]. A manager also needs to be aware when it is beneficial to add people to the team [McConnell 1996] and when not beneficial: as noted by Brooks [1995] time and manpower are not interchangeable. This component is important to practitioners from their development process perspective [Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Interviews 2001].

•

Developers believed they did a good job [Interviews 2001].

•

Project completed on time [Ginzberg 1981; DeMarco and Lister 1999; Jiang and Klein 2000.

•

Project completed within budget [Ginzberg 1981; DeMarco and Lister 1999; Jiang and Klein 2000].

•

There was a project plan [Interviews 2001].

•

Project was well planned [Interviews 2001].

•

Requirements were accepted by team as realistic/achievable [Interviews 2001].

•

Project met requirements of customer/users [Ginzberg 1981; DeMarco and Lister 1999; Jiang and Klein 2000].

There was an integrated project plan for this project. This component is part of a well-planned project, and includes both the presence of, and use of a project plan [Verner, et al 1999; Procaccino and Verner 2001b; Interviews 2001]. It is related to proper planning. The Standish Group [1995a] found that executive IT managers reported that 9.6% (fourth most cited) of projects considered to be a success had proper planning. Further, 8.1% (seventh most cited) of those projects considered to be impaired had a lack of planning. Practitioners have reported the importance of a project plan from their perspective of the development process [Procaccino and Verner 2001b; Interviews 2001]. Based on the results from Pilot Study #1, this component is used to partially define success from the practitioner’s

31

perceptive. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001b; Interviews 2001].

•

Developers believed they did a good job [Procaccino and Verner 2001b; Interviews 2001].

•

Project completed on time [Standish Group 1995a].

•

Project completed within budget [Standish Group 1995a].

•

There was a project plan [Procaccino and Verner 2001b; Interviews 2001].

•

Project was well planned [Procaccino and Verner 2001b; Interviews 2001].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001b; Interviews 2001].

•

Project met requirements of customer/users [Standish Group 1995a].

Overall, this project was well planned (related to people, technology, scheduling, etc). This component is part of a well-planned project, and includes both the presence and use of a good project plan [Procaccino and Verner 2001b; Interviews 2001]. Thorough proper planning includes up-front accounting for the time needed to adequately administrate and test the project [Keider 1974]. As noted earlier, the Standish Group [1995a] found that executive IT managers reported that 9.6% (fourth most cited) of projects considered to be a success had proper planning. Further, 8.1% (seventh most cited) of those projects considered to be impaired had a lack of planning. Practitioners have reported the importance of this variable from their perspective of the development process [Procaccino and Verner 2001b; Interviews 2001]. Based on the results of Pilot Study #1, this component is used to partially define success from the practitioner’s perceptive. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001b; Interviews 2001].

•

Developers believed they did a good job [Procaccino and Verner 2001b; Interviews 2001].

•

Project completed on time [Standish Group 1995a].

32

•

Project completed within budget [Standish Group 1995a].

•

There was a project plan [Procaccino and Verner 2001b; Interviews 2001].

•

Project was well planned [Pressman 1997; Procaccino and Verner 2001b; Interviews 2001].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001b; Interviews 2001].

•

Project met requirements of customer/users [Standish Group 1995a].

3.2

Customer/User Support and Participation

It is important if a project is to be completed successfully for customer/user participation to occur in close collaboration with other principle stakeholders (including project management and the development team) [Ewasi-Mensah 1997]. Further, user participation can help achieve success from the perspective of practitioners, as well as the perspective of the rest of the organization [Procaccino and Verner 2001a; Interviews 2001]. In general, user participation supports buy-in, which may contribute to the following outcomes: •

Less likelihood of inclusion of unnecessary system features [McKeen and Guimares 1997]. o More support for negotiation of functionality/design between the development team and users [McKeen and Guimares 1997].

•

More access by the development team to the company and unit in which the system will be implemented [McKeen and Guimares 1997].

•

More realistic expectations by users [McKeen and Guimares 1997].

•

Increased feeling of ownership among users [McKeen and Guimares 1997].

•

Increased likelihood that users will be tolerate any changes imposed through the implementation of the new system [McKeen and Guimares 1997; Choe 1998].

•

Higher user commitment to the ultimate success of the completed system [McKeen and Guimares 1997].

•

Higher overall user satisfaction with the completed system [Bailey and Pearson 1983; McKeen and Guimares 1997] and its impact on use of the system itself, as well as use of the system’s output [Bailey and Pearson 1983].

•

Higher overall utilization of system output by users [Bailey and Pearson 1983].

3.2.1

Customer/User Components

33

This section presents in detail software project development components related to Customer/User Support and Participation that were investigated for possible inclusion in this study.

Overall, customer/users had a high level of confidence in the project manager/development team. Research suggests that practitioners consider a high level of confidence by customer/users in the project manager and team to be important to the development process and their perspective of success [Verner, et al 1999]. Specifically, they have mentioned that if customer/users do not have confidence in project manager, customer/users might resist dealing with the project manager [Verner

et

al

1999],

presumably

hampering

communication

between

the

project

manager/development team and the customer/users. Further, there is some evidence to suggest that this component is also important to the organization/management’s perception of success [Procaccino and Verner 2001a]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a].

•

Developers believed they did a good job [Procaccino and Verner 2001a].

•

Project completed on time [Procaccino and Verner 2001a].

•

Project completed within budget [Procaccino and Verner 2001a].

•

There was a project plan [Procaccino and Verner 2001a].

•

Project was well planned [Procaccino and Verner 2001a].

•

Customers/users have realistic expectations regarding functionality [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Adequate communication between project manager/team and customer/users [Verner, et al 1999].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001a].

•

Project met requirements of customer/users [Procaccino and Verner 2001a].

The level of customer/users participation during the development process was high.

34

User participation is an important component for meeting various project objectives, particularly meeting requirements [Ginzberg 1981; McConnell 1996; Hunton and Beeler 1997; Tackett and Van Doren 1999; Verner, et al 1999]. This assertion is based on participative decision-making theory [Saleem 1996] and the TQM philosophy of stakeholder participation [Ravichandran and Rai 2000]. User participation has far reaching implications for the development process yet some research suggests that users are “rarely involved in product development” [Tackett and Van Doren 1999]. The Standish Group [1995a] found that executive IT managers reported that 15.9% (most cited) of projects considered to be a ‘success’ had user involvement (participation). Further, 12.8% (most cited) of those projects considered to be ‘challenged’ lacked user input (participation), as did 12.4% (second most cited) of those considered to be ‘impaired’. Practitioners have reported the importance of this component from their perspective of the development process [Lyytinen 1988; Verner, et al 1999]. Specifically, practitioners have mentioned the importance of being able to work closely with users, specifically with regard to requirements analysis [Verner, et al 1999]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•

Developers believed they did a good job [Jiang and Klein 2000; Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed on time [Glass 1996; Procaccino and Verner 2001a; Jiang and Klein 2000; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Procaccino and Verner 2001a; Jiang and Klein 2000; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•

Project was well planned [Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Participating customers/users stayed throughout the project [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Customers/users have realistic expectations regarding functionality [Ginzberg 1981; McKeen and Guimares 1997].

35

•

There was adequate communication between team and customer/users [Procaccino, et al 2002b].

•

Agreement on requirements between customer/users and practitioners [Ginzberg 1981].

•

Requirements were clear and complete (scope was well-defined) [McKeen and Guimares 1997; Choe 1998].

•


•

Requirements gathering resulted in well-defined software deliverables [McKeen and Guimares 1997; Choe 1998].

•

Project met requirements of customer/users [Ginzberg 1981; Procaccino and Verner 2001a; Lu and Wang 1997; Jiang and Klein 2000; Procaccino, et al 2002a; Procaccino, et al 2002b].

Although beyond the scope of system development (and this study), user participation has also been shown to positively contribute to a system’s ease of use and quality of information provided [Rainer and Watson 1995].

36

Participating customers/users stayed throughout the project. Research suggests that practitioners consider participating customer/users to be important to the development process and their perspective of project success [Verner et al 1999; Procaccino and Verner 2001a]. As noted earlier this assertion is based on participative decision-making theory [Saleem 1996] and the TQM philosophy of stakeholder participation [Ravichandran and Rai 2000]. Practitioners mentioned the importance of both users who may leave the organization and/or the departure of a specific customer or sponsor who may have been the only people actually interested in the project [Verner et al 1999]. Further, there is some evidence to suggest that this component is important to both practitioner’s and the organization/management’s perception of success [Procaccino and Verner 2001a]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a; Procaccino and Verner 2001b].

•


•

Project completed on time [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

There was a project plan [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project was well planned [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project met requirements of customer/users [Procaccino, et al 2002a; Procaccino, et al 2002b].

Customers/users had realistic expectations regarding system functionality. Ginzberg suggests that the perceptions of user expectations early in the development process are particularly important in determining the ultimate success of a given project [Ginzberg 1981]. Further, unrealistic expectations can lead to dissatisfied users, system avoidance and low system usage [Ginzberg 1981]. If practitioners assume that agreement with users had been reached

37

regarding project scope and functionality, and but in fact this has not occurred, users may be dissatisfied with the completed system [Ginzberg 1981]. As a result, customer/user expectations need to be managed, and not just left to chance [Rainer and Watson 1995].

User participation and adequate communications between the development team and customer/users supports realistic expectations among customer/users [McKeen and Guimares 1997; Ginzberg 1981]. Therefore, unrealistic expectations can increase the risk associated with successful software development [Moynihan 2000]. The Standish Group [1995a] found that executive IT managers reported that 8.2% (fifth most cited) of projects considered to be a ‘success’ had realistic expectations. Further, 5.9% (seventh most cited) of those projects considered to be ‘challenged’ had unrealistic expectations, as did 9.9% (fourth most cited) of those considered to be ‘impaired’. Customer/users with realistic expectations is related to customer/users understanding the functionality to be delivered by the system, and ultimately to meeting customer/user requirements. Practitioners have also noted the importance of this variable to their development process perspective of success [Lyytinen 1988; Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a; Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•

Project completed on time [Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•


•


38

•

There was adequate communication between team and customer/users [Ginzberg 1981].

•

Agreement on requirements between customer/users and practitioners [Ginzberg 1981].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001b; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


Customer/users provided feedback to the development team. Customer/user feedback is related to customer/user participation. As noted earlier, research suggests that practitioners consider participating customer/users to be important to the development process and their perspective of project success [Verner, et al 1999; Procaccino and Verner 2001b]. A component of customer/user participation is user feedback to the development team and practitioners value feedback from the user community [Verner, et al 1999]. Part of this feedback includes user testing of the developing system [Verner, et al 1999]. This component is also related to adequate communication between the development team and customer/users, and such communication supports effective requirements management [Pressman 1998]. Research suggests that this component is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Verner, et al 1999; Procaccino and Verner 2001b].

•

Developers believed they did a good job [Verner, et al 1999; Procaccino and Verner 2001b].

•

Project completed on time [McConnell 1996].

•

There was a project plan [Verner, et al 1999; Procaccino and Verner 2001b].

•

Project was well planned [Verner, et al 1999; Procaccino and Verner 2001b].

•

High level of customer/users participation during development [Ginzberg 1981].

•

Customers/users have realistic expectations regarding functionality [Ginzberg 1981].

•

Agreement on requirements between customer/users and practitioners [Ginzberg 1981; Pressman 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

39

•

Requirements were clear and complete (scope was well-defined) [McConnell 1996; Pressman 1998].

•

Requirements were accepted by team as realistic/achievable [Verner, et al 1999; Procaccino and Verner 2001b].

•

Requirements gathering resulted in well-defined software deliverables [McConnell 1996; Pressman 1998].

•

Team understood what customer/users wanted based on the requirements [Pressman 1998].

•

Stability of the requirements during the development process [McConnell 1996].

•

Project met requirements of customer/users [Pressman 1998].

There was adequate communication between the project manager/team and customer/users. The process of developing software is unique in its dependence on close collaboration between three major stakeholders: project management, IT staff and customer/users [Ewusi-Mensah 1997]. In fact, participants are said to typically engage in “intense collaboration” with other project stakeholders during the development process [Ewusi-Mensah 1997] (see Figure 2). The participation of practitioners is integral to both the process of developing software and the resulting product, as they are at the critical core of software development [Brooks 1995; DeMarco and Lister 1999], both in terms of what they do and with whom they interact (see Figure 2).

The development team must strive to keep users in the requirements development loop, as users are best able to determine the functionality required for a system [McKeen and Guimares 1997; Clavadetscher 1998]. This component is also related to user participation in the development process, and has been documented as an important variable [Standish Group 1995a; Clavadetscher 1998; Pressman 1998], particularly in the early stages of the development process [Ginzberg 1981]. A project manager needs to cultivate an atmosphere that supports communication between the development team and customer/users, which in turn will facilitate effective requirements management [Pressman 1998] and user participation. Adequate communication supports agreement between customer/users and the development team on the

40

project scope and functionality. Agreement is important because if not reached, users may be dissatisfied with the completed system [Ginzberg 1981].

There is also a need for negotiation skills in working with the customer and users on scheduling, budgeting, [Pressman 1998] and communicating the development team’s activities to the users [Keider 1974]. An example of the importance of this communication occurs when customers/users lack an adequate understanding of the technology that is used in a development project [Jiang, et al 2001]. As a result, they fail to grasp the implications of the technology, as they relate to the development process and their business objectives [Jiang, et al 2001]. To some extent, the technology may represent a ‘black box’ to the customers and users. Further, software practitioners often do not grasp neither customer/user’s perception of the utilized technology, or its implications for communicating with the customers/users [Jiang, et al 2001]. (Horst Rittel termed such a situation as a ‘symmetry of ignorance’ [Rittel 1984]) Practitioners have reported the importance of this variable from their perspective of the development process [Lyytinen 1988; Interviews 2001]. It also relates to practitioner’s perception of the importance of having other stakeholder participation in the decision-making process [Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Lyytinen 1988; Interviews 2001].

•

Developers believed they did a good job [Lyytinen 1988; Interviews 2001].

•

There was a project plan [Lyytinen 1988; Interviews 2001].

•

Project was well planned [Lyytinen 1988; Interviews 2001].

•

High level of customer/users participation during development [McConnell 1996; Ginzberg 1981].

•


•

Agreement on requirements between customer/users and practitioners [Boehm 1981; Ginzberg 1981; McConnell 1996; Pressman 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

41

•

Requirements were clear and complete (scope was well-defined) [Boehm 1981; McConnell 1996; Pressman 1998].

•

Requirements were accepted by team as realistic/achievable [Lyytinen 1988; Interviews 2001].

•

Requirements gathering resulted in well-defined software deliverables [Boehm 1981; McConnell 1996; Pressman 1998].

•

Team understood what customer/users wanted based on the requirements [Boehm 1981; Pressman 1998].

•

Stability of the requirements during the development process [Boehm 1981; McConnell 1996].

•

Project met requirements of customer/users [Boehm 1981; Clavadetscher 1998; Pressman 1998].

Customers/users made adequate time available for requirements gathering. Customer/users that make adequate time available for requirements gathering has implications for the quality and completeness of requirements [Glass 1998]. It is also related to customer/user participation [Verner, et al 1999; Standish Group 1995a]. Practitioners mentioned the importance of users making adequate time to work with the development team [Verner, et al 1999; Interviews 2001], including spending enough time to adequately define requirements [Interviews 2001].

McConnell suggests that requirements analysis and gathering can be an “easy target” to shortchange particularly when development schedule is tight because it does not result in any code generation, [McConnell 1996]. However, there can be a high cost associated with such a practice, as functionality needs to be added later [McConnell 1996]. The Standish Group [1995a] found that more than 30% of the projects sampled in their study had problems with requirements management. The 1995 European Software Process Improvement Training Initiative revealed that requirements specification and managing user requirements were the two biggest problems associated with software development [Leffingwell and Widrig 2000]. In short, inadequate requirements gathering can be found in most project failures [Glass 1998]. Research suggests that

42

this variable is causally linked (as an independent variable) to the following (dependent) variables:

3.3

•


•


•

Project completed on time [Glass 1998].

•

Project completed within budget [Glass 1998].

•


•


•


•

Agreement on requirements between customer/users and practitioners [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were clear and complete (scope was well-defined) [Procaccino, et al 2002b].

•

Requirements were accepted by team as realistic/achievable [Verner, et al 1999; Interviews 2001].

•

Project met requirements of customer/users [Glass 1998]. Requirements Management

Brooks [1995] suggests that in the early days of software development, practitioners were said to be building software. Now we speak of growing software, due to its level of complexity. He suggests that it is too complex "to be accurately [and completely] specified in advance and too complex to be built faultlessly" [Brooks 1995]. Related to this complexity, other research suggests that requirements determination is the most important step in software development [Zmud 1980; Rainer and Watson 1995] and in determining the ultimate value of the system [Boehm and Egyed 1998], as poor requirements can be found in most project failures [Glass 1998; Schenk and Vitalari 1998]. Requirements management includes relative changes to project scope and the understanding between the development team and customer/users regarding requirements and the functionality of the final product. Brooks also addresses the importance of requirements gathering and management on the entire software development process, and he

43

suggests that determining what to build is the most difficult part of software development and he adds the following [Brooks 1995]: “No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later. The hardest single part of building a software system is deciding precisely what to build… No other part is more difficult to rectify later… The most important function that software builders do for their clients is the iterative extraction and refinement of the product requirements.” Requirements analysis is one of the first steps in the software development process, with implications that extend throughout the entire project [Chatzoglou 1997; Boehm and Egyed 1998]. Further, there is mounting evidence that some of the most common and serious problems associated with developing software can be traced back to requirements and their management [Leffingwell and Widrig 2000]. Nidumolu [1996] suggests that requirements analysis can help alleviate some of the uncertainty associated with software development because it is “the most important of all [development] phases and has the greatest impact on future phases”. Therefore, adequate requirements management has far-reaching implications for effective and efficient software development.

Requirements management impacts the total cost of developing software, in terms of money, time and talent, as adequate requirements gathering and management helps to alleviate costly rework [Boehm and Basili 2001]. Problems related to inadequate requirements gathering and management, such as missing functionality, are considerably more expensive to correct later in the development process, compared to corrections made during initial requirements analysis and specification [Boehm 1981; McConnell 1996]. Corrections of system functionality made during the requirements phase can be five to ten times less than the costs associated with addressing these issues during the coding stage, and as much as twenty times less expensive when compared to fixing the problem during the maintenance phase [Leffingwell and Widrig 2000]. The costs incurred by these corrections include re-specification, redesign, recoding and retesting

44

[Leffingwell and Widrig 2000]. Excessive cost impacts organizational/managerial success criteria of on time and within budget completion. Inadequate requirements management also has implications for the extra effort required from practitioners [Leffingwell and Widrig 2000].

Adequate requirements gathering/management is an important consequence of a positive relationship between customer/users and the development team. This relationship must be conducive to a complete and accurate determination of requirements, as customers and users often do not know what they really need. (They may have some thoughts regarding what they want, but it cannot be assumed this is the same as their needs.) Further, they users are rarely experienced in requirements elicitation, particularly at the necessary level of detail [Brooks 1995]. Customer/users participation includes their participation in the development process, their confidence in the development team, their expectations, and their knowledge of their business needs and data.

3.3.1

Requirements Management Components

This section presents the development components related to Requirements Management that were investigated for possible inclusion in this study.

How

did

the

functional

scope

change

from

requirements

gathering

through

to

completion/abandonment? This component has implications for completing a project that meets customer needs within time and budgetary constraints. A project needs a clearly defined beginning and an end [Keider 1974], and well-defined requirements [Rainer and Watson 1995; McConnell 1996; Glass 1998]. Glass [2001a] suggests that unstable requirements and overly optimistic schedule estimates are the two most common causes of runaway projects (i.e. those projects that far exceed their budget and time

45

estimates, and fail to meet functionality needs). The list of requirements can grow dramatically from gathering to design, often 50 times larger than the original list [Glass 2001a]. In general, changing requirements specifications result in projects that are more difficult to manage [Nidumolu 1996].

The Standish Group [1995a] found that executive IT managers reported that 11.8% (third most cited) of projects considered to be ‘challenged’ had changing requirements, as did 8.7% (sixth most cited) of those considered to be ‘impaired’. Although not directly investigated in this study, clear and complete requirements can inhibit development lessons learned (from a practitioner and organizational perspective), as continuing requirements changes can throw a project into disarray, particularly in the absence of adequate change control. The Standish Group [1995a] also found that executive IT managers reported that 13.0% (third most cited) of projects considered to be a ‘success’ had clear statement of requirements. Further, 12.3% (second most cited) of those projects considered to be ‘challenged’ had incomplete requirements, as did 13.1% (most cited) of those considered to be ‘impaired’. Practitioners have reported the importance of this variable from their perspective of the development process [Interviews 2001]. They also mentioned this within the context of proper change control, particularly as it related to requirements [Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables:

•


•


•

Project completed on time [McConnell 1996; Glass 1998; Jiang and Klein 2000; Glass 2001a].

•

Project completed within budget [Glass 1998; Jiang and Klein 2000; Glass 2001a].

•


•

Project was well planned [Nidumolu 1996; Interviews 2001].

46

•


•

Requirements gathering resulted in well-defined software deliverables [Procaccino, et al 2002b].

•

Project met requirements of customer/users [Glass 1998; Jiang and Klein 2000; Glass 2001a]

Agreement on requirements was reached between customer/users and the development team: This component is related to adequate communication with customer/users and adequate requirements gathering. It is also related to customer/user participation in the development process, which is vital to complete requirements definition. This participation ideally takes place early in the development process [McKeen and Guimares 1997; Choe 1998; Clavadetscher 1998]. If practitioners assume wrongly that requirements agreement regarding project scope and functionality, has been achieved with users the result is that users are dissatisfied with the completed system [Ginzberg 1981]. As noted earlier, the development team must strive to keep users in the requirements development loop, as users have the best perspective to determine the appropriate functionality from the system [Clavadetscher 1998]. It is reasonable to expect such participation will result in agreement on requirements between customer/users and the development team. This variable is related to clear and complete requirements.

Agreement on requirements between customer/users and the development team is also related to customer/users understanding the functionality to be delivered by the system, customer/users having realistic expectations and ultimately meeting customer/user requirements. Practitioners have reported that agreement on requirements between customer/users and the development team is importance from their perspective of the development process [Procaccino and Verner 2001b]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


47

•


•

Project completed on time [Glass 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Glass 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

There was a project plan [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project was well planned [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project met requirements of customer/users [Boehm 1981; Glass 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

Requirements were clear and complete (scope of project’s functionality was well-defined). Research notes the importance of a well-defined scope, and well-defined requirements [Rainer and Watson 1995; McConnell 1996; Ewusi-Mensah 1997]. Customer/user participation helps to more completely and accurately define user requirements, and this results in a better overall understanding of the system to be developed [McKeen and Guimares 1997]. Clear and complete requirements support the development of a project that meets customer needs [Nidumolu 1996; Glass 1998; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

As noted earlier, the Standish Group [1995a] found that executive IT managers reported that 13.0% (third most cited) of projects considered to be a ‘success’ had clear statement of requirements. Further, 12.3% (second most cited) of those projects considered to be ‘challenged’ had incomplete requirements, as did 13.1% of those considered to be ‘impaired’. This factor also relates to changing requirements and change control. Practitioners have reported the importance of this factor from their perspective of the development process [Lyytinen 1988; Standish Group 1995a; McConnell 1996; Verner, et al 1999; Procaccino and Verner 2001a; Procaccino and Verner 2001b; Interviews 2001]. Further, there is some evidence to suggest that this variable is

48

also important to the organization/managements’ perception of success [Procaccino and Verner 2001a]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•

Project completed on time [McConnell 1996; Nidumolu 1996; Glass 1998; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Nidumolu 1996; Glass 1998; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•

Project was well planned [Nidumolu 1996; Procaccino and Verner 2001a; Procaccino and Verner 2001b; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•


•

Requirements gathering resulted in well-defined software deliverables [Procaccino, et al 2002b].

•

Project met requirements of customer/users [Boehm 1981; Nidumolu 1996; Glass 1998; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

49

Requirements were accepted by the development team as realistic/achievable. Requirements will not be seen as realistic by the development team if customer/users continually make changes, particularly after requirements have been baselined [McConnell 1996]. Such practices can hinder a project from ‘rapid development’ (i.e. not meeting scheduling and presumably budgeting estimates) [McConnell 1996]. This component, at least in part, is related to negotiating skills of the project manager and development team [Pressman 1998]. Practitioners have reported the importance of this component from their perspective of the development process [Interviews 2001]. Based on the results from Pilot Study #1, this variable is used to partially define success from the practitioner’s perceptive. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•

Project completed on time [Boehm 1981; McConnell 1996].

•

Project completed within budget [Boehm 1981; McConnell 1996].

•


•

Project was well planned [Nidumolu 1996; Interviews 2001].

•


•

Project met requirements of customer/users [Boehm 1981; McConnell 1996].

Requirements gathering resulted in well-defined software deliverables. Clear and complete requirements result in well-defined software deliverables. Deliverables may include system functionality, screens, reports and system utilities etc. Practitioners have reported the importance of this factor from their perspective of the development process [Verner, et al 1999; Procaccino and Verner 2001a; Interviews 2001]. Further, there is evidence to suggest that this factor is important to both practitioner’s and the organization/management’s perception of success [Procaccino and Verner 2001a]. Research suggests that this component is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a].

50

•

Developers believed they did a good job [Procaccino and Verner 2001a].

•

Project completed on time [Nidumolu 1996].

•

Project completed within budget [Nidumolu 1996].

•

There was a project plan [Procaccino and Verner 2001a].

•

Project was well planned [Procaccino and Verner 2001a].

•

A high confidence level in the project manager/team members was held by the customer/users [Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001a].

•

Project met requirements of customer/users [Boehm 1981].

The size/complexity of the project negatively impacted requirements gathering. There is evidence that project size/complexity can hamper requirements gathering, resulting in unclear, incomplete and/or unstable requirements [Glass 1998; Procaccino and Verner 2001b]. Practitioners have reported the importance of this development process component [Procaccino and Verner 2001a; Interviews 2001]. Practitioners mentioned the difficulty of managing changes to large projects with many requirements [Procaccino and Verner 2001a; Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Developers had a sense of achievement [Procaccino and Verner 2001a; Interviews 2001].

•

Developers believed they did a good job [Procaccino and Verner 2001a; Interviews 2001].

•

Project completed on time [Glass 1998].

•

Project completed within budget [Glass 1998].

•

There was a project plan [Procaccino and Verner 2001a; Interviews 2001].

•

Project was well planned [Procaccino and Verner 2001a; Interviews 2001].

•

Requirements were clear and complete (scope was well-defined) [Glass 1998; Procaccino and Verner 2001b].

•

Requirements were accepted by team as realistic/achievable [Procaccino and Verner 2001a; Interviews 2001].

•

Requirements gathering resulted in well-defined software deliverables [Glass 1998; Procaccino and Verner 2001b].

51

•

Team understood what customer/users wanted based on the requirements [Glass 1998].

•

Project met requirements of customer/users [Procaccino and Verner 2001b].

The development team understood what customer/users wanted based on gathered requirements. This component is related to adequate communication between the development team and customer/users. Users may be dissatisfied with the completed system without this understanding [Ginzberg 1981]. Understanding what customer/users want based on requirements is also related to clear and complete requirements. This impacts how well the project is planned and managed, as well as to what extent it meets customer/user requirements [Nidumolu 1996]. Further, research suggests that practitioners are a motivated group, but it is important that they are given specific objectives in order to use this motivation effectively and efficiently [McConnell 1996]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Project completed on time [McConnell 1996].

•

Project completed within budget [McConnell 1996].

•

Project was well planned [Nidumolu 1996].

•

High level of customer/users participation during development [McConnell 1996; Ginzberg 1981].

•


•

Agreement on requirements between customer/users and practitioners [Boehm 1981; Ginzberg 1981; McConnell 1996; Pressman 1998; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Requirements were clear and complete (scope was well-defined) [Boehm 1981; McConnell 1996; Nidumolu 1996; Pressman 1998].

•

Requirements gathering resulted in well-defined software deliverables [Boehm 1981; McConnell 1996; Pressman 1998].

•

Stability of the requirements during the development process [Boehm 1981; McConnell 1996].

•

Project met requirements of customer/users [Boehm 1981; Nidumolu 1996; Clavadetscher 1998; Pressman 1998]. Overall, requirements were stable during the development process.

52

This component is related to clear and complete requirements. The development process can be better controlled, managed, measured and studied for future best practices when requirements are relatively stable [Nidumolu 1996]. This factor has important implications for both practitioner and organizational/managerial perceptions of success, as it involves changing requirements during the development process. As noted earlier, Glass [2001a] suggests that unstable requirements and overly optimistic schedule estimates are the two most common causes of runaway projects. The original list of requirements at design phase can grow to be 50 times larger by the time the project reaches the requirements phases [Glass 2001a]. As noted earlier, the Standish Group [1995a] found that executive IT managers reported that 11.8% (third most cited) of projects considered to be ‘challenged’ had changing requirements, as did 8.7% (sixth most cited) of those considered to be ‘impaired’. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•

Project completed on time [McConnell 1996; Nidumolu 1996; Glass 1998; Jiang and Klein 2000; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Nidumolu 1996; Glass 1998; Jiang and Klein 2000; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•


•


•


•


•

Requirements gathering resulted in well-defined software deliverables.

•

Project met requirements of customer/users [Boehm 1981; Glass 1998; Jiang and Klein 2000; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

53

Requirements of customer/users were met. This component is widely cited as a component of organizational/managerial perception of project success [Keider 1974; Pinto and Slevin 1988; Standish Group 1995a; Jones 1995; Clavadetscher 1998; Baccarini 1999; Linberg 1999; Leffingwell and Widrig 2000; Wohlin, et al 2000]. Therefore, this component is used as part of the definition of success from the organizational/managerial perceptive. Practitioners have reported the importance of this variable from their development process perspective [Interviews 2001]. Research suggests that this variable is causally linked (as an independent variable) to the following (dependent) variables: •


•


•


•


•


The next chapter presents details of the main study, including its particular focus and perspective, anticipated outcomes and the importance of the study.

54

4. Main Study: Overview This study will provide a greater understanding of some of the components of the software development process leading to cancelled, or projects that are delivered late and over-budget, and/or do not meet customer/user requirements [Barki, et al 1993; Jones 1995; Standish Group 1995a; McConnell 1996; Glass 1998; Verner, et al 1999; Jiang and Klein 2000; Ravichandran and Rai 2000]. Because challenged projects can leave software practitioners de-motivated and professionally unfulfilled. Our research seeks to focus management attention on the importance of a number of early components of the software development process. Downstream problems will be avoided or lessened if these components receive more managerial attention As noted earlier, our focus will be on project success from the perspective of software practitioners.

The remainder of this section is organized as follows. Firstly, a brief discussion of the focus of this study is presented, including its emphasis on practitioners and the process of the software development process (as opposed to the software product). This is followed by a discussion of the working definition of project success used in this study. Next, we discuss the anticipated outcomes of the study. The importance of the study and its associated research models follows. Details of the data collection instrument are presented next. The research questions and research hypotheses follow.

4.1

Project Focus

This section provides a brief discussion of the focus of this study. Boehm [1981], Brooks [1995] and McConnell [1996] have suggested that effective and efficient software development can be framed in terms of the following four components: •

People (project staff assignment, team organization, motivation).

•

Process (managerial and technical methodologies, including implications for rework avoidance, quality assurance, development fundamentals, risk management, resource targeting, lifecycle planning, customer orientation).

55

•

Technology (hardware and software tools).

•

Product (size and characteristics).

Technological components directly associated with developing software, including hardware, software and specific development methodologies, will not be addressed in this study as the study focus is (1) project stakeholders (primarily software practitioners) and, (2) aspects of the software development process.

4.1.1

Project Stakeholders

Software practitioners are the focus of the study and the source of its data because they play such a critical role within the development process and their perspective has not been widely explored in the research literature. It follows then, that this study’s definition of success is based on the perspective of software practitioners. However, practitioners are also asked to evaluate the relative success of the project from an organizational/managerial perspective. This organizational perspective is captured through the timeliness and affordability of the product (compared to schedule and budgeting estimates), and level of functionality required by the customer/user. The project’s senior/executive management and/or customers and users may be more knowledgeable about these particular components than are practitioners. However, we believe that practitioner views are important and have a significant impact on the eventual success or otherwise of the project. In addition, components of customer/user support and participation are explored because of their early and on-going impact on the development process.

4.1.2

Process vs. Product

As noted earlier, the focus of this study is on various people-related, non-technical components of the software development process. Emphasis is placed on some of process components that have

56

an early and on-going impact on software development, including those related to sponsor/management support and participation, customer/user support and participation, and requirements management. Non-technical components that have an impact particularly early in the development process are especially important, as software practitioners report that software development problems stem more from differing expectations among project stakeholders, which originate at the outset of the project, than from events that happen along the way [Glass 1998].

The resulting software product is not directly addressed in this study, except to serve as a barometer for the relative success of the development process from an organizational/managerial perceptive (i.e., the project was completed on time, within budget and met customer/user requirements).

4.2

Defining Project Success

As illustrated by Chapter 2, defining software project success is a complicated undertaking. Further, although McConnell [1996], Linberg [1999], and Procaccino and Verner [2001c], among others, investigated project success from the perceptive of software practitioners, there is not much other research that addresses this topic. Further, the limited research done thus far, suggests that the generally accepted organizational/managerial definition of project success is not an accurate reflection of software practitioners’ perception of project success (i.e. they have a different perception of what constitutes a successful project from the organization and management as a whole) [Linberg 1999; McConnell 1996; Procaccino and Verner 2001c]. For example, research indicates that meeting budget and time estimates is not necessarily important to practitioners [Linberg 1999; McConnell 1996; Procaccino and Verner 2001c]. As a result, there is a need to develop and use a dependent success variable that can be included in a causal model. Pilot Study #1 (see below) was conducted in order to investigate components of software

57

development, and to derive a working definition of project success. As noted earlier, practitioners are asked to evaluate project success from their own perspective and they are also asked to evaluate project success according to the following criteria, which represents the organizational/managerial perspective: •

The project was delivered on time.

•

The project was delivered within budget.

•

The project met customer/users requirements.

We are then able to examine what correlations exist between the two perspectives for the same projects. We next present a more detailed discussion of the pilot study used to derive the working definition of project success.

4.2.1

Pilot Study #1: Software Practitioner’s Success Components

A survey, which included a maximum of 54 project variables (see Appendix D), was developed through literature review and interviews with practitioners (see Appendix F) in order to assess those components of the development process that most impact practitioners’ perception of project success. The interviews were driven by the following questions: •

Think of projects that YOU (not management, customers or users) considered to be a success. Why do you consider it to be successful?

•

Think of projects that YOU (not management, customers or users) considered to be a failure. Why do you consider it to be a failure?

Data for this pilot study was collected between March and May 2001. A total of 43 industry professionals enrolled in Drexel University’s College of Information Science & Technology’s online and traditional classes responded to the survey. Each survey question began with the phrase, “It is important to your perception of project success that…”. Responses were given on a fivepoint scale, ranging from ‘Agree’ (5) to ‘Disagree’ (1). Our major aim was to discover the following:

58

What are the five most important components of software development projects that practitioners consider important to their perception of project success (pilot study survey questions 2.01 through 2.54)? The mean score for each variable was calculated by summing the responses to each of the questions (5, 4, 3, 2 or 1) and then dividing this sum by the number of responses (see Appendix E for complete descriptive statistics). This calculation can be expressed by the following equation: Score = Total score of all responses / number of respondents The success variables were sorted by descending means. The variables with the ten highest means are shown in Table 1. The number of respondents (N) varied among some of the questions because new questions were added to the instrument and as a result, not all respondents had the opportunity to respond to all 54 questions.

Table 1: Top 10 Highest Ranked Success Variables Rank 1 2 3 4 5 6 7 8 9 10

Variables Q2.05 Project had a plan Q2.03 Well planned Q2.34 You did a good job Q2.02 Good management practices Q2.23 You had a sense of achievement Q2.19 Requirements are accepted by the development team Q2.18 Requirements can be clarified Q2.04 Developers provide feedback Q2.40 Team is skilled Q2.20 Requirements are clear and understood

N Mean 12 4.75 12 4.67 43 4.53 12 4.50 43 4.44 12 4.42 12 4.33 12 4.33 12 4.33 12 4.33

Standard Deviation 0.45 0.65 0.63 0.80 0.50 0.67 1.23 1.15 0.98 0.89

The ten variables included in Table 1 were then ranked by ascending coefficient of variation (COV), which provides some measure of the ‘stability’ of the calculated means. The calculation can be expressed by the following equation: COV = Standard deviation / mean

59

The variables with the five lowest coefficient of variation, which are included in Table 2, were used to form a working definition of software practitioner project success. The five variables that are included in the index variable are represented by the following survey questions. The variables comprising the working project success definition are listed below in order of ascending coefficients of variation. It is important to your perception of project success that there is a project plan. [Standish Group 1995a; Verner, et al 1999; Procaccino and Verner 2001b], It is important to your perception of project success that you have a sense of achievement while working on a project. [Boehm 1981; McConnell 1996; Glass 1999; Procaccino and Verner 2001b; Procaccino and Verner 2002]. It is important to your perception of project success that you do a good job (i.e. delivered quality) while working on a project. [Glass 1999; Procaccino and Verner 2002; Procaccino and Verner 2001b], It is important to your perception of project success that the project is well planned. [Standish Group 1995a, Pressman 1998; Verner, et al 1999; Procaccino and Verner 2001b]. It is important to your perception of project success that requirements are accepted by the development team as realistic/achievable. [McConnell 1996; Pressman 1996; Pressman 1998; Procaccino and Verner 2001b].

Table 2: Success-Related Project Variables With Lowest C.O.V. Rank 1 2 3 4 5

4.3

Variables Q2.05 Project had a plan Q2.23 You had a sense of achievement Q2.34 You did a good job Q2.03 Project was well planned Q2.19 Requirements accepted by development team

N Mean 12 4.75 43 4.44 43 4.53 12 4.67 12 4.42

Standard Coefficient of Deviation Variation 0.45 0.10 0.50 0.11 0.63 0.14 0.65 0.14 0.67 0.15

Importance of Study

The importance of this study can be discussed from two perspectives: (1) theoretical and (2) practical.

(1) Theoretical importance: This study will help to fill the current quantitative, survey-based research gap (from the perception of software practitioners) in the causal impact of early and on-

60

going non-technical components of software development,. To date, software project management research has largely been qualitative and anecdotal in nature [Rainer and Watson 1995; Pinto and Mantel 1990]. Research has included the recounting of ‘war stories’ with associated lessons learned [Brady and DeMarco 1994; Glass 1998; McConnell 1996; Hohmann 1999; Jiang and Klein 2000] and assorted ‘best practices’ from both organizational and managerial perspectives [Humphreys 1990; Brady and DeMarco 1994; Brooks 1995; McConnell 1996; Pressman 1998; Hohmann 1999; Glass 2001a]. Within this general stream of research are found attempts to improve the quality of the software product by improving the software development process, largely though methodologies intended to make tit repeatable and more predictable [Curtis, et al 2001]. These methodologies include, among others, Capability Maturity Model (CMM), the Software Capability Maturity Model (SW-CMM), the People Capability Maturity Model (P-CMM), Software Process Improvement and Capability dEtermination (SPICE), and Extreme Programming (XP).

(2) Practical importance: This study’s findings will prove useful to organizations that develop software. One of the main purposes of this study is to raise awareness among project managers of the potential downstream impact of their actions (or inactions) during the development process. The impact includes various aspects of the resulting software product, including its timeliness of delivery, affordability and ability to meet customer/user requirements. (The impact of these actions will be quantitatively illustrated through standardized Betas, R, R-square and probabilities.) Raising awareness among project stakeholders is important to software development because of its implications for the effective and efficient application of limited organizational resources: time, talent and money. This study is also intended to help make the development process more repeatable, resulting in better project estimation and planning.

61

The analysis will provide some insight into which of the project development components addressed has the greatest impact on project success. Further, managers can use the insight gained to evaluate the status of an on-going development project while it is still early enough to take corrective action, regardless of the methodology used. Jones notes that projects that may be “in serious trouble” [i.e., at risk for not meeting stakeholder objective(s)] “are not (normally) identified until very late in development” [Jones 1995]. Boehm [1991] notes the following: “…problems would have been avoided or strongly reduced if there had been an explicit early concern with identifying and resolving their [project] high-risk elements. Frequently, these projects were swept along by a tide of optimistic enthusiasm during their early phases that caused them to miss some clear signals of high-risk issues that proved to be their downfall later.” As outlined earlier, the results from this study can assist managers in evaluating their on-going projects, and improve managerial decision-making as lessons learned are applied to other software development projects. Further, organizations should have an interest in understanding the key components of software development that practitioner’s value. These components have a strong effect on practitioner motivation, and motivation has been reported to have a huge impact on productivity [Boehm 1981; McConnell 1996; Boehm 1999].

To our knowledge, the only previously survey instrument used in a study of this type was produced by Verner and Cerpa [1999] and reported in Procaccino and Verner [2001a], Procaccino, et al [2002a and 2002b; see Appendix G and H], and Procaccino and Verner [2001c]. Our study builds on these previous studies through additional literature review and a pilot study (see Appendices L and M), The data collected will be used to evaluate and create a causal chain of variables.

4.4

Anticipated Outcomes

62

The main outcome of this study is a quantitatively based causal model that includes some of the early and on-going components that affect the success of the software development process. This model will be depicted through ordinal regression analysis and a Bayesian Belief Network (BBN). Each technique will assist in determining the relative importance of each of the components of system development addressed in the study.

Unlike regression techniques, which are based on a normal distribution, Bayesian models make no assumption regarding data distribution. Further, they incorporate an element of chance through general randomness. As such, they can be thought of as an extension of statistical, rule-based models [Fenton and Neil 2000a]. Fenton and Neil suggest that BBN are “by far the best solution” for modeling risk assessment [Fenton and Neil 2000a]. BBNs are useful in illustrating the hypothetical effect of changes in various combinations of explanatory variables on the probability of project success through ‘what-if’ analysis. Figure 3 presents a conceptual depiction of the qualitative (dependencies among independent and dependent variables) and quantitative (calculated probabilities) aspects of Bayesian models through a graphic representation.

Figure 3: Conceptual components of Bayesian Belief Networks

The next chapter presents the research models, research questions and research hypotheses for this study.

63

64

5. Research Models, Questions and Hypotheses

This section discusses the theoretical and empirical research models that are used to structure the research in this study. The research questions and hypotheses are also presented, each of which has direct implications for this study’s causal model.

5.1

Theoretical Research Model

The theoretical model for this study (see Figure 4) presents a meta- (high-) level perspective of the construction of the final (Bayesian) model for this study. Figure 4 represents a model that is theoretical at this stage in that no results of any data collection are considered. In other words, the theoretical model represents the overall methodology that is used in the study to construct the final model. (Depicting the inclusion of data into the framework of the study is shown through an empirical model that is discussed in the sub-section, 5.2 Empirical Research Model.) The model begins with variables that represent specifically identified components of the early stages of the software development process. (The final survey instrument contains these components.) These variables were identified through literature review, pilot studies and interviews with software practitioners. Variables are related to sponsor/management support and participation, customer/user support and participation, and requirements management, which represent the major project categories within the survey instrument. Also identified and tested were correlations (relationships) among specific variables, which were largely derived from literature review. It is these correlations that help to form the causal relationships within the model, which is the final piece of the theoretical model (Bayesian Belief Network). The ultimate (final) dependent variable within the Bayesian model is overall project success from the perspective of software practitioners.

65

Figure 4: Theoretical Research Model

5.2

Empirical Research Model

The empirical research model is included below (Figures 5), which essentially introduce the collected data as it is applied and analyzed through the theoretical framework previously mentioned and shown in Figure 4. This empirical model represents a meta-level perspective of the quantitative analysis that was performed on the project components mentioned in the discussion of the theoretical model (as mentioned previously, components are categorized based on the organization of the survey instrument, i.e., sponsor/management support and involvement components, customer/user support and participation and requirements management). The first part of the empirical model graphically depicts the regression analysis (Section 8.2.3: Ordinal Regression Analysis) of each of the causal relationships within the final Bayesian Belief Network. The purpose of this analysis is to investigate the relative explanatory power (measured through an R2 value) of each of the causal ‘sub’-models within the overall Bayesian model. The analysis tests the collected data against specific proposed causal relationships (among independent/dependent variables) suggested by literature review (Chapter 3: Non-Technical Components of Software Development).

The second part of the empirical model (Figure 6) graphically depicts the testing of the collected data against the specifically proposed correlations (relationships) that were derived from literature

66

review. Statistically significant correlations (Section 8.2.2: Bivariate Correlation Analysis), combined with this evidence from literature review (Chapter 3: Non-Technical Components of Software Development), provide evidence of (validate) the causal relationships that are the ‘building blocks’ of the proposed Bayesian model.

Figure 5: Empirical Research Model

5.3

Research Questions

The following research questions are to be addressed by this study after responses to applicable survey questions have had their responses recoded so higher numbers represent positive responses. All question numbers refer to the survey instrument. Each of the arrows in the following associated figures begin at an independent variable and end (point to) a dependent variable. RQ1.

Is there a significant positive correlation between having a committed sponsor/champion, users that made adequate time available for requirements gathering and well-defined software deliverables, and a high level of confidence by the customer/users in the development team? (See Figure 6.)

Figure 6: Correlations For Research Question 1

67

RQ2.

Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, having a committed sponsor/champion and users that made adequate time available for requirements gathering, and a high level of customer/user participation? (See Figure 7.)


RQ3.

Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project and a high level of customer/user participation, and participating customer/users who stayed throughout the project? (See Figure 8.)


RQ4.

RQ5.

Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, and users that made adequate time available for requirements gathering? (See Figure 9.)

Figure 9: Correlations For Research Question 4 Is there a significant positive correlation between a high level of customer/user participation, users that made adequate time available for requirements gathering and clear/complete requirements (well-defined project scope), and agreement on requirements between customer/users and practitioners? (See Figure 10.)

68


RQ6.

Is there a significant positive correlation between a high level of customer/user participation and users that made adequate time available for requirements gathering, and clear/complete requirements (well-defined project scope)? (See Figure 11.)


RQ7.

Is there a significant positive correlation between a high level of customer/user participation, changes in the functional scope (negative correlation) and clear/complete requirements (well-defined project scope), and well-defined software deliverables? (See Figure 12.)


69

RQ08. Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, a high level of confidence by the customer/users in the development team, participating customer/users who stayed throughout the project, agreement on requirements between customer/users and practitioners, well-defined software deliverables, and practitioners’ overall perception of project success? (See Figure 13.)


RQ09. Is a high level of customer/users participation during development the most important variable in predicting whether a project will be considered a success by practitioners? 5.4

Support For Research Questions

The following section briefly presents the basis for asking each of the nine (9) research questions. References of (independent) refer to independent variables in the causal model and references to (dependent) refer to dependent variables.

RQ1 was asked in order to provide evidence that would support a causal relationship between having a committed sponsor/champion (independent), users that made adequate time available for requirements gathering (independent) and well-defined software deliverables (independent), and a high level of confidence by the customer/users in the development team (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from

70

Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

RQ2 was asked in order to provide evidence that would support a causal relationship between having a sponsor/champion that stayed throughout the project (independent), having a committed sponsor/champion (independent) and users that made adequate time available for requirements gathering (independent), and a high level of customer/user participation (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners. In addition, this question was also asked due to the nature of the relationship between customer/user who made adequate time for requirements gathering and a high level of customer/users participation. Specifically, the former is assumed to be a component of the latter.

RQ3 was asked in order to provide evidence that would support a causal relationship between having a sponsor/champion that stayed throughout the project (independent) and a high level of customer/user participation (independent), and participating customer/users who stayed throughout the project (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

71

RQ4 was asked in order to provide evidence that would support a causal relationship between having a sponsor/champion that stayed throughout the project (independent), and users that made adequate time available for requirements gathering (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

RQ5 was asked in order to provide evidence that would support a causal relationship between a high level of customer/user participation (independent), users that made adequate time available for requirements gathering (independent) and clear/complete requirements (well-defined project scope) (independent), and agreement on requirements between customer/users and practitioners (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

RQ6 was asked in order to provide evidence that would support a causal relationship between a high level of customer/user participation (independent) and users that made adequate time available for requirements gathering (independent), and clear/complete requirements (welldefined project scope) (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the

72

relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

RQ7 was asked in order to provide evidence that would support a causal relationship between a high level of customer/user participation (independent), changes in the functional scope (independent) and clear/complete requirements (well-defined project scope) (independent), and well-defined software deliverables (dependent), as there is evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners. The independent variable, changes in the functional scope, is expected to have a negative correlation with well-defined software deliverables because of the nature of the coding of the independent variable. A higher degree of change in the functional scope was coded higher on a five-point scale (from “Got much smaller” to “Got much larger”) and agreement that software deliverables were indeed well-defined equates to a higher response on a five-point scale. As a result, it is expected that as the degree of functional change within a project goes up, the likelihood that respondents will agree that software deliverables for that project were well-defined will tend to go down.

RQ8 was asked in order to provide evidence that would support a causal relationship between having a sponsor/champion that stayed throughout the project (independent), a high level of confidence by the customer/users in the development team (independent), participating customer/users who stayed throughout the project (independent), agreement on requirements between customer/users and practitioners (independent), well-defined software deliverables (independent), and practitioners’ overall perception of project success (dependent), as there is

73

evidence in the literature that this variable has implication for project success from the perspective of practitioners. This question will also help to verify the preliminary findings from Procaccino and Verner [2001b] through analyzing the relationship of this variable to this study’s working definition of project success from the perspective of practitioners.

RQ9 was asked because customer/user participation had the most extensive list of cited casual (dependent) components of any of the investigated project components. As a result, customer/user participation has the most interactions (combining independent and dependent relationships) with other variables within the BBN.

5.5

Hypotheses

The hypotheses to be addressed by this study include the following: HO1.

There is a significant positive correlation between having a committed sponsor/champion, users that made adequate time available for requirements gathering and well-defined software deliverables, and a high level of confidence by the customer/users in the development team.

HO2.

There is a significant positive correlation between having a sponsor/champion that stayed throughout the project, having a committed sponsor/champion and users that made adequate time available for requirements gathering, and a high level of customer/user participation.

HO3.

There is a significant positive correlation between having a sponsor/champion that stayed throughout the project and a high level of customer/user participation, and participating customer/users who stayed throughout the project.

HO4.

There is a significant positive correlation between having a sponsor/champion that stayed throughout the project, and users that made adequate time available for requirements gathering.

HO5.

There is a significant positive correlation between a high level of customer/user participation, users that made adequate time available for requirements gathering and clear/complete requirements (well-defined project scope), and agreement on requirements between customer/users and practitioners.

HO6.

There is a significant positive correlation between a high level of customer/user participation and users that made adequate time available for requirements gathering, and clear/complete requirements (well-defined project scope).

HO7.

There is a significant positive correlation between a high level of customer/user participation, changes in the functional scope (negative correlation) and

74

clear/complete requirements (well-defined project scope), and well-defined software deliverables. HO8.

There is a significant positive correlation between having a sponsor/champion that stayed throughout the project, a high level of confidence by the customer/users in the development team, participating customer/users who stayed throughout the project, agreement on requirements between customer/users and practitioners, well-defined software deliverables, and practitioners’ overall perception of project success.

HO9.

A high level of customer/users participation during development is the most important variable in predicting whether a project will be considered a success by practitioners.

The next chapter discusses the statistical methods, data instrument and data analysis for the main study.

75

6. Main Study: Data Collection Instrument

This section presents details of the survey instrument used in this study, including the organizations asked to participate, how they were contacted, how data was collected, explanation of the demographic variables and a brief discussion of the various response scales.

6.1

Industry Contacts

The respondents are software practitioners, including programmers, database developers and system analysts. During December 2001 and January 2002, e-mails were sent to 168 organizations listed in the 2000-2001 Eastern Technology Council Annual Directory asking for their participation in this study. Member organizations that developed software were selected based on their biographies. These organizations were advised that they would be contacted later in the Winter (and subsequently in the Spring), pending the finalization of the survey instrument.

In addition, a national database of 3,714 software project managers from an equal number of organizations was acquired from Applied Computer Research, Inc. (Phoenix, Arizona; www.acrhq.com). Applied Computer Research supplied a list that included the data fields listed in Appendix M. The following lists summarized demographics of the organizations that were contacted for this study: •

Organizations from all fifty States of the Union, plus the District of Columbia.

•

Organizations with more than 15 information technology employees.

•

12 identified industries, plus “Other”.

•

About 13% of the organizations were Fortune 1000 firms.

•

About 12% of the organizations were Fortune 500 Public firms.

•

About 7% of the organizations were Fortune 500 Private firms.

•

About 4% of the organizations were InformationWeek 500 firms.

76

The file was imported into Microsoft Excel and the following steps were taken. Organizations included in the national mailing were selected as follows: 1. Sorted 3,714 records by SYSMFR (computer systems manufacturer). 2. Removed records with missing SYSMFR (=“ ”). 3. Sorted 3,671 remaining records by STATE. 4. Removed Puerto Rico (STATE=PR). 5. Sorted 3,665 remaining records by GENDER. 6. Removed records where can’t determine male or female contact (GENDER=U). 7. Defined RANDOM# as random number generated for 3,640 remaining records. 8. Records sorted by ascending random number. 9. Selected first 2,000 records in final record list. This number was chosen based on an estimated 5-10% organization response rate. Edits made to selected records are included in Appendix N.

6.2

Distribution Mechanism

Between April 26 and May 1, 2002, a letter was mailed (as e-mail addresses were not available) to each of the software project managers of the selected organizations. Organizations were instructed

to

ask

members

of

their

software

development

teams

to

go

to

http://129.25.27.178/survey.htm.

A Web-based survey is appropriate for the targeted sample because software practitioners are presumed to have a connection to the Internet and the necessary computer hardware. They are also expected to have adequate skills to utilize a graphical user interface (Microsoft Windows) to respond to the survey through a Web browser. Also, the survey was designed with ‘generic’ HTML coding in order to be as compatible as possible with a wide range of browsers and computer hardware. Compatibility was tested through piloting the Web site with several people at various organizations. 6.3

Survey Instrument

77

The instrument used in this study is based on several surveys previously used in Procaccino and Verner [2001a], Procaccino and Verner [2001c], Procaccino, et al [2002a] and Procaccino, et al [2002b].

An analysis of inter-correlations between variables is used to help determine appropriate candidate variables for inclusion in a causal model. Evidence of causal relationships between specific variables investigated was included with citations in Chapter 3. These relationships are intended to provide project managers with insight into some of the risks that can threaten the development process, as well as the resultant product. These relationships, in contrast, are not intended to evaluate the product after the project has been completed or abandoned [Ginzberg 1981; Procaccino and Verner 2001; Procaccino, et al 2002a; Procaccino, et al 2002b; Procaccino, et al 2002b]. Managers need to be aware of the necessity of removing non-technical obstacles that can inhibit their development team’s ability to work. Managers need to be skilled at shielding the staff from administrative concerns (scheduling, budgeting and tracking), acting as buffers between those doing work and concerns related to all other project activities [Pressman 1998]. This approach should assist in investigating cause and effect of these project variables [Ginzberg 1981].

The survey instrument includes components of both software process and product from the perspective of practitioners, but has a strong emphasis on process. Overall, the literature suggests that well-managed, successful, development projects are more likely to be perceived as being successful by all stakeholders, including end-users. Management of the development process can have both a long and short-term effects. Project management impacts the current project (shortterm) and subsequent projects whose processes are influenced from lessons learned during the current project (long-term). Additionally, the relative success of the product that was most

78

recently developed can have an immediate effect on practitioners, and a longer-term impact on the end-users who use it on a daily basis. Finally, customers who paid for the work are also impacted.

The variables included in Survey Sections 3, 4 and 5 (Sponsor/Management Support and Participation and Your Project, Customer/User Support and Participation, and Requirements Management, respectively) were derived from the variables discussed in Chapter 3. The remainder of this chapter includes a discussion of the variables categorized under Respondent Demographic (Survey Section 1), General Project Variables (Survey Section 2) and Project Success (Survey Section 6).

6.3.1

Respondent Demographic Variables

The survey variables grouped under the survey section, Your Background (included in Table 3), are described below. Their general intent is to provide insight into the relative appropriateness of the respondents.

Table 3: Variables From Survey Section 1 Variable Q1.01 Q1.02 Q1.04

Variable Description/Operational Definition Gender Age range Years as practitioner

Measure nominal nominal ratio

Length 2 5 5

79

1.01 Indicate your gender. This variable partially describes the survey respondent. It is also included for investigation of possible correlation with other variables, with implications for subsequent studies.

1.02 Indicate your age range. This variable partially describes the respondent and it helps to validate the appropriateness of the respondent to answer this survey. It is also included for investigation of possible correlation with other variables, with implications for subsequent studies.

1.03 How many years have you been a software practitioner? This variable partially describes the respondent and it helps to validate the appropriateness of the respondent to answer this survey. It is also included so that we can investigate possible correlation with other variables, with perhaps implications for subsequent studies. Table 3 shows the variables, definitions and data details for Survey Section 1: Your Background.

6.3.2

General Project Variables

A few of the survey variables that are grouped under the Survey Section 2: Your Project are described below. (However, we do not expect many of these variables to have specific correlations to variables in the Sponsor/Management Support and Participation, Customer/User Support and Participation, and Requirements Management sections of the survey)

Q2.02 How many IT people participated in developing this project? This variable partially describes the developing organization and project team. It is also included for investigation of possible correlations with other variables, with implications for subsequent studies.

80

Q2.04 Did you have a financial interest in the organization that developed this project? This variable has implications for considering a project a success from an organizational/financial perspective. A respondent who has a financial interest in the company developing the software may be more likely to consider that project to be a success from the organizational perspective if that project led to more professional work for the organization.

Q2.05 What was your responsibility(s) on this project? This variable partially describes the survey respondent and it helps to validate the appropriateness of the respondent to answer the survey. It is also included for investigation of possible correlation with other variables, with implications for subsequent studies.

Q2.12 This project had approximately how many function points? The number of function points serves as a partial measure of application size/complexity, which has implications for other components of project development, including overall project planning and estimating (with added uncertainty associated with added complexity [Pressman 1997; Pressman 1998] and the meeting of organizational objectives. The number of function points also has implications for practitioners’ perception of project success, as practitioners have mentioned the difficulty in managing functional changes to large projects with many requirements [Procaccino and Verner 2001a; Interviews 2001]. There is evidence that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Project completed on time [Jones 1995; Pressman 1997; Jiang and Klein 2000].

•

Project completed within budget [Pressman 1997; Jiang and Klein 2000].

•

Project was well planned [Pressman 1997].

•

Project met requirements of customer/users [Jiang and Klein 2000; Procaccino and Verner 2001a; Interviews 2001].

Q2.13 This project had approximately how many source lines of code (SLOC)?

81

The number of source lines of code serves as a partial measure of application size/complexity, which has implications for other components of project development, including overall project planning, estimating [Pressman 1997; Pressman 1998] and the meeting of organizational objectives. The number of source lines of code also has implications for practitioners’ perception of project success, as they have mentioned the difficulty in managing functional changes to large projects with many requirements [Procaccino and Verner 2001a; Interviews 2001]. There is evidence that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Project completed on time [Jones 1995; Pressman 1997; Jiang and Klein 2000].

•

Project completed within budget [Pressman 1997; Jiang and Klein 2000].

•

Project was well planned [Pressman 1997].

•

Project met requirements of customer/users [Jiang and Klein 2000; Procaccino and Verner 2001a; Interviews 2001].

Q2.14 You had a sense of achievement while you worked on this project. Practitioner’s sense of achievement has implications for their motivation, which has a direct impact on productivity [Boehm 1981; McConnell 1996]. Productivity, in turn, impacts the developing organization’s ability to complete the project on time and within budget [Procaccino, et al 2002a; Procaccino, et al 2002b]. Practitioners have indicated that this variable contributes to their perception of project success [Boehm 1981; McConnell 1996; Glass 1999; Procaccino and Verner 2002]. Based on the results from Pilot Study #1, this variable is used to partially define success from the practitioner’s perception. There is evidence that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Project completed on time [Boehm 1981; McConnell 1996; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Boehm 1981; McConnell 1996; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project met requirements of customer/users [Boehm 1981; McConnell 1996;

82

Procaccino, et al 2002a; Procaccino, et al 2002b]. Q2.15 You did a good job while working on this project. Practitioner’s belief that they are doing a good job has implications for practitioner motivation, which has a direct impact on productivity [Boehm 1981; McConnell 1996]. Productivity, in turn, impacts the developing organization’s ability to complete the project on time and within budget [Procaccino, et al 2002a; Procaccino, et al 2002b]. Practitioners have indicated that this variable contributes to their perception of project success [Glass 1999; Procaccino and Verner 2001b; Procaccino and Verner 2002]. Based on the results from Pilot Study #1, this variable is used to partially define success from the practitioner’s perceptive. There is evidence that this variable is causally linked (as an independent variable) to the following (dependent) variables: •

Project completed on time [Boehm 1981; McConnell 1996; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project completed within budget [Boehm 1981; McConnell 1996; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Project met requirements of customer/users [Boehm 1981; McConnell 1996; Procaccino, et al 2002a; Procaccino, et al 2002b].

Q2.16 This project was completed on time. Completing a project within scheduled time is widely cited as a component of organizational/managerial perception of project success [Keider 1974; Pinto and Slevin 1988; Standish Group 1995a; Jones 1995; Clavadetscher 1998; Baccarini 1999; Linberg 1999; Leffingwell and Widrig 2000; Wohlin, et al 2000].

The Standish Group [1995a] found that of those projects that were either completed late or cancelled, greater than one out of three had schedule overruns between 200 and 300% and the average for all of their 8,380 sampled projects was 222% of original budget. As much as twothirds of all software development projects are “grossly” late in completion [McConnell 1996].

83

Q2.17 This project was completed within budget. Completing a project within budget is widely cited as a component of organizational/managerial perception of project success [Keider 1974; Pinto and Slevin 1988; Standish Group 1995a; Jones 1995; Clavadetscher 1998; Baccarini 1999; Linberg 1999; Leffingwell and Widrig 2000; Wohlin, et al 2000]. Therefore, this variable is used to partially define success from the organizational/managerial perceptive. Also included in this composite definition is completing the project within schedule estimates and meeting all user requirements.

The Standish Group [1995a] found that of those projects that were either completed late or cancelled, almost one of three had cost overruns between 150 and 200% and the average for all of their 8,380 sampled projects was 189% of original budget.

The next chapter includes elements of data analysis, including research questions and a discussion of several aspects of validity and reliability.

84

7. Main Study: Data Analysis

This chapter is organized as follows. Details of the research questions and the associated proposed independent/dependent variable relationships are presented first. Next, several aspects of the validity and reliability of the study are discussed, followed by details of processing of the raw data that was collected from the online survey. Descriptive statistics of organizational and individual respondent demographics conclude the chapter.

7.1

Research Questions and Independent/Dependent Variables

Table 4 formally relates the research questions to independent and dependent survey variables, and the associated statistical method.

Table 4: Research Questions and Independent/Dependent Variables Research Question and Associated Variables RQ01. Is there a significant positive correlation between the following variables: • having a committed sponsor, • customers/users made adequate time for requirements gathering, • requirements gathering resulted in well-defined software deliverables, and customer/users having a high level of confidence in the project manager/development team and the following variables? RQ02. Is there a significant positive correlation between the following variables: • having a sponsor/champion throughout the project, • having a committed sponsor, • customers/users made adequate time for requirements gathering, and high customer/user participation in the development process?

Independent Variable(s) Q3.03 Q4.08 Q5.05

Dependent Variable(s) Q4.01

Statistical Method Correlation Analysis (Spearman’s) at α < 0.05

Q3.02 Q3.03 Q4.08

Q4.02

Correlation Analysis (Spearman’s) at α < 0.05

85

Table 4 (cont’d) Research Question and Associated Variables RQ03. Is there a significant positive correlation between the following variables: • having a sponsor/champion throughout the project, • high customer/user participation in the development process, and participating customers/users stayed throughout the project? RQ04. Is there a significant positive correlation between the following variables: • having a sponsor/champion throughout the project, and customers/users made adequate time for requirements gathering? RQ05. Is there a significant positive correlation between the following variables: • high customer/user participation in the development process, • customers/users made adequate time for requirements gathering, • requirements were clear and complete (scope was well-defined), and agreement on requirements reached between customer/users and development team? RQ06. Is there a significant positive correlation between the following variables: • high customer/user participation in the development process, • customers/users made adequate time for requirements gathering, and requirements were clear and complete (scope was well-defined)? RQ07. Is there a significant positive correlation between the following variables: • high customer/user participation in the development process, • changes in the functional scope, • requirements were clear and complete (scope was well-defined), and requirements gathering resulted in well-defined software deliverables? RQ08. Is there a significant positive correlation between the following variables: • having a sponsor/champion throughout the project, • customer/users having a high level of confidence in the project manager/development team, • participating customers/users stayed throughout the project, • agreement on requirements reached between customer/users and development team, • requirements gathering resulted in well-defined software deliverables, and practitioners’ overall perception of project success?

Independent Variable(s) Q3.02 Q4.02

Dependent Variable(s) Q4.03

Statistical Method Correlation Analysis (Spearman’s) at α < 0.05

Q3.02

Q4.08


Q4.02 Q4.08 Q5.03

Q5.02


Q4.02 Q4.08

Q5.03


Q4.02 Q5.01 Q5.03

Q5.05


Q3.02 Q4.01 Q4.03 Q5.02 Q5.05

Q6.01


Table 4 (cont’d)

86

Research Question and Associated Variables RQ09. Is high customer/user participation the most important variable in predicting whether a project will be considered a success by practitioners, according to the BBN model?

7.2

Independent Variable(s) Q4.02

Dependent Variable(s) Various

Statistical Method Probabilitybased what-if analysis based on Bayesian theory

Validity and Reliability

In general, an analysis of validity accesses how well the utilized research instrument investigates the concepts of interest. Validity cannot be proven for a given study, but it can be assessed in relation to the validity of the particular utilized measures, as well as the findings and conclusions. Face, criterion and content validity relate to the employed measures, while internal and external validity can provide support for an overall assessment of the results and conclusions [Babbie 2001]. In addition, construct validity has implications for the measures, findings and conclusions. Validity supports, and leads to, reliability, which addresses repeatability of the study findings.

Face validity is assessed “on its face” (surface) [Babbie 2001]. It is achieved for this study by a survey instrument that was developed through interviews with software practitioners, relevant literature review and piloting of the survey instrument. Criterion-based (or predictive) validity “is based on external criterion” [Babbie 2001]. The three major sections of the instrument, Management/Sponsor Support and Participation, Customer/Users Support and Participation, and Requirements Management, have been widely documented in the literature as representing important early components of the software development process (see Chapter 3: Non-Technical Components of Software Development). Principal component factor analysis and bivariate correlation analysis lend evidence, albeit indirectly, to criterion-related validity through construct validity (see Chapter 8: Analysis and Findings). Content validity is a measure of “how much a measure [variable] covers the range of meanings included within a concept” [Babbie 2001]. This

87

is expected to be acceptably high, as the questions were derived from interviews, literature review and pilot testing.

Internal validity refers to how well conclusions drawn from results “accurately reflect what has gone on in the experiment itself”. This includes any inadvertent stimulus that may act as a confounding variable upon the dependent variable. Specifically, this includes accessing how well our identified independent variables (non-demographic) influence practitioner’s overall perception of project success. The results of Pilot Study #2 provide an acceptable level of internal validity for the component makeup and use of our working definition of practitioners’ perception of project success. The independent variables included in this study’s survey instrument (related to management/sponsor support and participation, customer/user support and participation, and requirements management) are valid predictors of project success because they were based on interviews with practitioners, extensive literature review and piloting of questions. External validity addresses if it reasonable to expect that the findings produced through the survey instrument are generalizable to the sampled population, and not merely an artifact of the study. The sample for this study was gathered through a professional directory and a national database of software project managers (see Section 6.1: Industry Contacts). Practitioner’s perception of project success was based on Pilot Study #2, which investigated specific aspects of software development that practitioners consider important to their perception of project success (Appendix E-1 and E-2). Particular considerations of external validity include the following: •

Sample: Our sample is relatively random, as respondents reported doing professional software development across several industries, project sizes, project types and States of the Union. The only incentive offered to potential participants is our willingness to share the final results.

•

Ecological: The survey instrument is expected to provide realistic relevant information, and insight into addressing our hypothesizes because the instrument was developed through interviews with practitioners (see Appendix F), relevant literature review and piloting of the survey instrument.

•

Replication: Findings from this study should be replicable, as a random

88

representation of practitioners helped to ensure that results of correlation analysis are not attributable to mere chance. •

Researcher and/or subject (post-test) effects. By thorough analysis and development of our survey and pilot test, we have been able to identify and correct any researcherbased effects (bias) within the final survey instrument. We do not anticipate any other researcher-based effects largely because our respondents will complete the survey quite independent of our research team. No threats from subject effects are expected to be introduced because of this independence and the assurance that their responses will be analyzed with complete confidentiality with regard to both themselves and their organizations.

We believe that this study has a reasonable level of reliability due to the expected validity of the survey instrument and sampling methodology. Additional measures that can help to assure reliability are focus groups, test/retest, split-half (asking two groups of respondents different versions of questions to check for high inter-group correlation) and intercoder (one person records observations, a second person records a proportion of the observations, then there is a check of correlation between the two coders). Although this last technique is not appropriate for this study due to the use of electronic data collection, the others may be helpful for future work (see Chapter 10: Recommendation For Futher Study).

7.3

Processing of The Raw Data

A total of 346 records were collected through the Web site, including 168 usable responses. The following cases were removed prior to data analysis. •

164 test cases were collected both before and during data collection in order to ensure that the Web-based data collection mechanism was working correctly. These test cases were identified by non-valid codes (i.e., codes that did not match any code that had been provided to legitimate respondents).

•

One case was removed as it included only respondent demographics, as the respondent noted that the survey was not applicable to his or her particular organization.

•

Six (6) cases were removed because the respondents indicated that the projects they were referring to had not yet been completed for various reasons and the focus of this study is completed projects.

89

•

Seven (7) cases were removed because the respondents indicated that they had only managerial responsibilities on their project (i.e., they did not have any role as a practitioner).

Table 5 summaries the survey respondents and response rate for this study. Since participating project managers were instructed to request that members of their development staff complete the survey, it is not possible to determine a practitioner response rate.

Table 5: Summary of Organizations and Respondents Providing Usable Responses

Total Contacted Incorrect Address*** Net Contacts Organizations Responding Organizational Response Rate

Eastern Technology Council* 168 16 152 6 3.95%

Applied Computer Research** 2,000 37 1,963 123 6.27%

Totals 2,168 54 2,114 129 6.10%

*

Includes 123 organizations contacted via e-mail and 45 contacted via 1st-class mail. ** All contacted by 1st-class U.S. mail after being randomly selected from 3,714 records. *** Includes postal address or e-mail address.

Survey questions that did not require a numeric responses that had a missing response (values), response of “Don’t know” or response of “Not applicable” (all referred to as ‘irregular responses’) need to be coded as missing values. Such values are omitted from the final analysis. Variables with the response, “Don’t know”, were considered as missing values due to a lack of sufficient information to accurately and reliably recode these into the other available responses. Table 6 shows how irregular responses were automatically recoded by the survey Web site. Any missing values in numeric fields, such as Q1.03 Number of years as a software practitioner, were hand coded as “99”.

90

Table 6: Automatic Recording of Irregular Responses Original Response Missing responses Don’t know Not applicable

Automatically Recoded As 99 98 97

The following two additional fields were added to the final table: • •

“Month/Year” of each response. Percent of total people who worked on the project (Q2.02) that were full-time employees (Q2.03).

Appendix N summarized the various changes and recodings that were completed for each applicable survey variable. Variables that are not included not require any recoding. After all recoding was completed, the responses to each variable were imported into SPSS and frequencies were run (in SPSS: Analyze Menu, Descriptive Statistics, Frequencies option) to ensure valid responses and appropriate recoding.

7.4

Respondents and Organizational Demographics

The following list is a summary of the respondent demographics. (See Appendix P for frequency of responses to each survey question): Respondents: • 74% of the 168 respondents were male and 26% were female. (An unpublished 2001 Annual Report by the Bureau of Labor Statistics indicated that 73% of computer programmers and computer systems analysts/scientists were male.) •

About 45% of respondents reported being between 40-49 years of age. The next largest group reported being between 50-59 years of age (about 24%). 71% of respondents were 49 years old or less. (In 2001, the Bureau of Labor Statistics reported that 76% of computer programmers and computer systems analysts/scientists were 44 years age or less.)

•

The mean number of years of experience as a software practitioner was about 19 and the median was 20 (with a minimum of 1 and maximum of 42).

•

About 91% of respondents reported having no financial interest (ownership) in the developing organization.

•

Most respondent project responsibility included project manager/leader (46%), senior manager (27%) and/or programming analyst (27%). Respondents could have more than

91

one responsibility on a given project and any respondents who reported only managerial responsibilities on a given project were not included in this study. Development Organizations: • Most of the 126 responding organizations were in the manufacturing/service industry (40%) or the education industry (18%). •

The mean number of IT employees at the responding organizations was 85 and the median was 50 (with a minimum of 15 and maximum of 1,000).

Project: • Development work was conducted in 35 States, plus the District of Columbia. •

The States with the five highest percentage of work were Virginia (11%), Ohio (10%), North Carolina (7%), Illinois (7%) and New Jersey (6%).

•

The mean number of people (fulltime, part-time and consultants) who worked on the project was about 18 and the median was 6 (with a minimum of 1 and maximum of 400).

•

The mean number of fulltime people who worked on the project was about 14 and the median was 5 (with a minimum of 1 and maximum of 300).

•

About 72% of projects were new development efforts, about 25% were enhancements to existing projects and about 3% were maintenance of existing projects.

•

About 70% of projects were intended for an in-house ‘customer’, about 16% for both an in-house and outside customer and about 14% for outside customer.

•

About 95% of projects were completed and 5% were completed with reduced functionality.

•

The mean number of months to project completion was about 16 and the median was 12 (with a minimum of 1 and maximum of 73).

•

Most of the customers were business (58%) or government (20%).

•

Most developed systems were management information/business applications (74%) or ecommerce (15%).

•

The mean number of project function points was about 451 and the median was 10 (with a minimum of 3 and maximum of 10,000).

•

The mean number of project source lines of code was about 845,802 and the median was 20,000 (with a minimum of 75 and maximum of 10,000,000).

Respondents’ gender, age range and years of experience were each tested as a potential source of bias through Chi-square analysis with practitioner’s overall perception of project success [Jiang and Klein 2000]. There was no evidence of a significant relationship between any of these demographic variables and project success, which is an indication that no such bais was introduced by the measured characteristics of the respondents.

92

The next chapter presents details of the analysis and findings of the study, including overviews of the analytical methods and results of each of these methods, including the final proposed model.

93

8. Main Study: Analysis and Findings

This chapter contains a brief overview of each of the analytical methods used in this study, as well as the findings associated with each of these methods. Brief explanations of Cronbach’s alpha, principal component factor analysis, bivariate correlation and ordinal regression are presented first. Next, the results of each of these analyses are discussed. An overview of probability-based Bayesian Belief Networks, and this study’s Bayesian model, are presented next. A summary of important results conclude this section.

8.1

Overview of Analytical Methods

This sub-sections includes brief review of Cronbach’s alpha/factor analysis, bivariate correlation, ordinal regression analysis and Bayesian Belief Networks.

8.1.1

Cronbach’s Alpha/Factor Analysis

Cronbach’s alpha, used in conjunction with principal component factor analysis (discussed below), can help evaluate the relative internal consistency of a particular set of variables from the survey instrument. Essentially, internal consistency can be measured by evaluating how well the included variables collectively measure a single (unidimensional) latent construct by examining the average inter-item (variable) correlation. Cronbach’s alpha is a function of the number of variables and the average inter-correlation among these variables. The critical threshold for a ‘high’ internal reliability among a group of variables is at least 0.60 [Ellis and Webster 1998 cite Bagozzi and Yi 1988; Gefen, et al 2000 cite Nunnally 1967] or 0.70 [Santos 1999; Gefen, et al 2000 cite Nunnally 1978, and Nunnally and Bernstein 1994].

Principal Component factor analysis is a multivariate statistical method that can provide evidence

94

of the degree of multi-dimensionality among a particular group of variables through clustering of correlation coefficients (factor loadings) [Hair, et al 1992]. High factor loadings (correlation coefficients) among a group of variables provides evidence of convergent and discriminate validity among those variables. While Cronbach’s alpha provides an overall measure of internal consistency, factor analysis offers a more detailed analysis (i.e., regarding the amount of multidimensionality) of each individual latent construct. Factor loadings between 0.30 and 0.40 are considered to be “significant”, while loadings greater than 0.40 are considered to be “more important” [Hair, et al 1992].) Loadings of at least 0.19 and 0.26 can be thought of as significant at the 0.01 and 0.05 levels, respectively, provided the sample size is 100.

8.1.2

Bivariate Correlation Analysis

In general, correlation is a bivariate measure of the relative (linear) strength of association between two (2) variables. This study includes polychoric correlations, which evaluate the relationship between two ordinal variables. Correlation (represented by a correlation coefficient) is the ratio of “observed covariance of two standardized variables divided by the highest possible covariance when their values are arranged in the best possible match by order” (www2.chass.ncsu.edu/garson/pa765/correl.htm). A correlation coefficient indicates the relative strength and direction of the relationship between two variables. The strength of the relationship is measured from no association (0) to perfect correlation (1) and the direction of the relationship is shown by the sign (- or +) of the coefficient. Therefore, the coefficient can range between –1 and +1. Correlation does not assume variables to be either independent or dependent. As mentioned previously, a causal relationship between two variables depends on presence of the following [Gefen, et al 2000 cite Cook and Campbell 1979]: •

Correlation/association: when a cause event happens, a likely effect event will also happen (determined through correlation and/or regression analysis).

•

Temporal ordering, or precedence, of variables: cause event happens before the effect event (determined though experience, experiment and/or literature review).

95

•

Isolation of causal influences: rule out other un-modeled, plausible variables that could have influenced a dependent variable (determined though experience, experiment and/or literature review).

As a result, correlation analysis cannot provide direct evidence of the causal nature between the two variables. It can only provide evidence to support an association between two variables. However, the results of correlation analysis can be combined with practitioners’ experience, experiment and literature review to suggest specific causality between variables. The proposed relationships from such evidence are depicted graphically in the Bayesian Belief Network model shown in Figure 18.

8.1.3

Ordinal Regression Analysis

The model proposed in this study includes variables that have been measured on an ordinal scale. Regression analysis would be useful in order to explore some of the predictive relationships within the model. However, one of the main assumptions of ordinary least-squares linear regression is that variables are measured on a continuous scale (which also implies equal distance between data points and a meaningful zero point), which is not the case with the variables included in the model. Other assumptions, including variables that are normally distributed and exhibit a constant variance among dependent variable across all values of independent variables (homoscedasticity), are not necessarily met with ordinal variables [Hannah and Quigley 1996]. As a result, ordinal regression is more appropriate for analyzing the model, as it contains a series of polytomous discrete, ordinal dependent variable (such those based on a Likert scale).

8.1.4

Bayesian Belief Networks

Bayesian Belief Networks (BBN; also known as Belief Networks, Causal Probabilistic Networks, Causal Nets, Graphical Probability Networks, Probabilistic Cause-Effect Models and Probabilistic Influence Diagrams) are “directed acyclic graph” based on “conditional

96

independence” [Fenton and Neil 2000b]. A BBN is a graphic technique for displaying and identifying probabilistic relationships among independent and dependent variables. BBNs can be quickly updated, manipulated and re-initiated to support ‘what-if’ analysis, including the isolation of the impact of specific variables. They can represent complex structure through a graphical interface, as opposed to a series of formulas and text [Agena 2002b]. A completed BBN represents a model of a critical roadmap for decision support through the use of the chain rule to calculate joint probability distributions [Agena 2002b]). The strength of BBNs comes from their ability to include uncertainty (unknown information) in decision support models (reflected in correspondingly higher levels of uncertainty) to explore previously undetermined relationships among variables, as well as helping to describe these new relationships [Niedermayer 1998]. This ability contrasts with the traditional frequentist approach of probability-based reasoning, which requires known (observed) frequencies/probabilities of past events [Agena 2002a]. Since BBNs present their findings in the form of probabilities, the relative certainty of a variable taking on a particular value is explicit, unlike, for example, typical regression-based models, which utilize confidence intervals around parameter estimates. Further, regression models “lack any causal structure”, which is critical to adequately support decision-making [Fenton and Neil 2000a].

BBN’s contain nodes, their associated conditional probability tables and arcs. A node represents a variable. A conditional probability table (CPT) is associated with each node. Each of these tables contain a probability for each and every state (value) of the variable being represented, given the state(s) of its parent (independent) nodes. As a result, the sum of all probabilities within each node (variable) must be 1 (100%) because every possible state of that variable must be represented. The probabilities associated with each dependent node are based on the conditional probabilities of the particular combined state of each parent node [Fenton and Neil 2000a; Agena 2002b]. These probabilities can come from empirical data and/or expert opinion. (This study

97

relies solely on empirical data.) Through propagation, all conditional probabilities (dependent on ‘parent’ nodes) are recalculated when any of the associated probabilities are changed. Figure 14 shows a partial conditional probability table for the variable Q5.03, which has Q4.02 and Q4.08 as independent (parent) variables. As a result, the table must account for every combination of independent/dependent variables, given that each of the three variables can take on the following values: 1, 2, 3, 4 and 5 (Q4.02 is shown as equaling only 1, 2, 3 and 4 due to space limitations). It should be noted that each column (composite state of all three variables) in the table must add to 1 (100%).

Figure 14: Sample Conditional Probability Table From Hugin System

The final graphical component of a BBN is a directional arc, which is used to represent causal relationships between sets of variables [Fenton and Neil 2000a]. The arc begins at an independent variable and points to a dependent variable.

BBNs have some limitations. As is the case with any analytic model, the results produced by BBNs are only as useful as the information (probabilities) that are feed into it (inputs) [Niedermayer 1998]. BBNs are also not useful in handling new requests, such as those based on variables that are not already included in the model [Niedermayer 1998]. Further, BBNs are not particularly useful for modeling “political, financial, environmental and technical criteria” [Fenton and Neil 2000a. However, BBNs can be used in conjunction with Multicriteria Decision Aids (MCDA) to overcome this deficit, as MCDAs can incorporate these ‘external’ variables into probabilistically structured decision-making process [Fenton and Neil 2000a].

98

8.2

Results of Analytical Methods

This sub-section includes results of each of the analytic methods described in Section 8.2: Results of Analytical Methods, including Cronbach’s Alpha/factor analysis, analysis of bivariate correlation, analysis of ordinal regression and construction/use of a Bayesian Belief Network.

8.2.1

Cronbach’s Alpha/Factor Analysis

Latent constructs investigated include management/sponsor support and participation, customer/user support and participation, and requirements management. The following steps were used to evaluate the relative internal reliability of the management/sponsor, customer/users and requirements management sections of the survey instrument, as well as an overall evaluation of all three sections: 1. Cronbach’s alpha was calculated for each section of the survey instrument through an investigation of all variables within each section. A relatively low Cronbach’ alpha value (∝ < 0.70) for a given section of the survey instrument might be an indication that the variables within that section are multi-dimensional. (Factor loadings for any variable below 0.40 [see Section 8.1.1: Cronbach’s Alpha/Factor Analysis] were dropped from the analysis to help show any underlying structure of each latent construct. Default SPSS parameters were used when running the analysis.) 2. A principal components factor analysis was conducted for each section to investigate the multi-dimensionality among variables. Fewer identified latent constructs were an indication of a lower degree of multi-dimensionality (and a presumably higher level of internal reliability as calculated in Step 1 above). Relevant SPSS parameters for conducting Principal Component factor analysis included the following: •

Extraction method: Principal Component Analysis.

•

Rotation Method:

Varimax with Kaiser Normalization, which Hair, et al [1992] suggest is an effective rotation for simplifying the structure of the resulting factor loadings.

•

Missing values:

Replaced with means.

3. Cronbach’s alpha was calculated for each of the identified constructs from the previous step in order to determine level of internal consistency of each construct. 4. Each construct from Step 3 was identified and labeled. 5. Steps 1 through 4 were run for survey Sections 3, 4 and 5 combined as an additional

99

examination of internal consistency. Survey Section 3: Management/Sponsor Support and Participation: 1. Overall Cronbach’s alpha (for all variables included in this section) was relatively high (see below). Number of Cases 149

Number of Variables 9

Cronbach’s Alpha 0.84

2. Factor analysis: Tables 07 and 08 represent the results of the factor analysis conducted for Survey Section 3, Sponsor/Management Support and Participation. Three (3) constructs were identified, which explained about 85% of the total variance among all seven (7) variables. Eliminated

variables

included

Q3.06,

“The

project

manager(s)

was

knowledgeable/experienced overall in the application area”, and Q3.07, “The project manager(s) supported the development team”. 3. Cronbach’s alpha for each identified construct from factor analysis (Step 2): The resulting Cronbach’s alpha for each of constructs are shown in Table 8. 4. The constructs were labeled as follows: •

Component 1 label: “Sponsor”

•

Component 2 label: “Project planning”

•

Component 3 label: “Project manager”

Table 7: Total Variance Explained (Sponsor/Management)

Component 1 2 3 4 5 6

Rotation Sums of Squared Initial Eigenvalues Loadings % of % of Total Variance Cum. % Total Variance Cum. % 3.27 46.77 46.77 2.66 38.06 38.06 1.63 23.35 70.12 1.70 24.30 62.36 1.01 14.40 84.52 1.55 22.16 84.52 0.46 6.51 91.02 0.31 4.45 95.48 0.21 3.02 98.49

100

7

0.11

1.51

100.00

Table 8: Rotated Component Matrix (Sponsor/Management) Component Variable Q3.03 Q3.02 Q3.01 Q3.09 Q3.08 Q3.05 Q3.04

Question Sponsor committed Sponsor throughout Had sponsor Project was well-planned Had project plan Project manager throughout Had project manager

1 0.94 0.94 0.90

2

3

Construct Alpha 0.95

0.89 0.89

0.80 0.91 0.80

0.74

Notes: Rotation converged in 5 iterations. Factor loadings below 0.40 eliminated and highest loadings for each variable (row) considered.

Survey Section 4: Customer/User Support and Participation: 1. Overall Cronbach’s alpha (for all variables included in this section) was relatively high (see below). Number of Cases 147



101

2. Factor analysis: Tables 09 and 10 represent the results of the factor analysis conducted for Survey Section 4, Customer/User Support and Participation. Two (2) constructs were identified, which explained about 65% of the total variance among all seven (7) variables. Eliminated variables included only Q4.01, “Customer/users had a high level of confidence in the project manager/development team”. 3. Cronbach’s alpha for each identified construct from factor analysis (Step 2): The resulting Cronbach’s alpha for each of constructs are shown in Table 10. 4. The constructs were labeled as follows: •

Component 1 label: “Customer participation”

•

Component 2 label: “Problems with customers”

Table 9: Total Variance Explained (Customer/User)

Component 1 2 3 4 5 6 7

Initial Eigenvalues % of Total Variance Cum. % 3.38 48.24 48.24 1.17 16.73 64.97 0.71 10.19 75.16 0.61 8.73 83.89 0.48 6.81 90.71 0.33 4.77 95.48 0.32 4.52 100.00

Rotation Sums of Squared Loadings % of Total Variance Cum. % 3.12 44.54 44.54 1.43 20.44 64.97

Table 10: Rotated Component Matrix (Customer/User) Component Variable Q4.02 Q4.08 Q4.07 Q4.03 Q4.06 Q4.05r Q4.04r

Question Participation was high Customers made adequate time Adequate communications Stayed throughout project Customer provided feedback Problems due to number of customers Did not have realistic expectations

1 0.86 0.77 0.77 0.75 0.71

2

Construct Alpha 0.85

0.85 0.71

0.47

Notes: Rotation converged in 3 iterations. Factor loadings below 0.40 eliminated and highest loadings for each variable (row) considered. Survey Section 5: Requirements Management:

102

1. Overall Cronbach’s alpha (for all variables included in this section) was relatively high (see below). Number of Cases 149



2. Factor analysis: Tables 11 and 12 represent the results of the factor analysis conducted for Survey Section 5, Requirements Management. Two (2) constructs were identified, which explained about 57% of the total variance among all nine (9) variables. The first identified constructs had a relatively high alpha value (0.85). However, the second construct had a relatively low alpha value (0.10). This construct included Q5.01, “Functional scope changed” and Q509, “Requirements of customer/users were met”, which would seem to be negatively correlated. This makes intuitive sense as an increase in functional project scope makes it more difficult to meet requirements of customer/users. No variables needed to be eliminated from the analysis. 3. Cronbach’s alpha for each identified construct from factor analysis (Step 2): The resulting Cronbach’s alpha for each of constructs are shown in Table 12. 4. The constructs were labeled as follows: •

Component 1 label: “Clear, stable and understood requirements”

•

Component 2 label: “Project scope and meeting requirements”

103

Table 11: Total Variance Explained (Requirements Management)

Component 1 2 3 4 5 6 7 8 9

Initial Eigenvalues % of Total Variance Cum. % 4.01 44.60 44.60 1.07 11.91 56.51 0.95 10.55 67.06 0.74 8.24 75.30 0.69 7.62 82.92 0.45 4.97 87.90 0.42 4.64 92.53 0.35 3.93 96.46 0.32 3.54 100.00

Rotation Sums of Squared Loadings % of Total Variance Cum. % 4.01 44.55 44.55 1.08 11.96 56.51

Table 12: Rotated Component Matrix (Requirements Management) Component Variable Q5.03 Q5.05 Q5.07 Q5.04 Q5.02 Q5.08r Q5.06r Q5.01n Q5.09

Question Requirements clear and complete Well-defined deliverables Team understood customer wants Requirements accepted Agreement on requirements Requirements were unstable/volatile Size of project negative impact Functional scope change Requirements were met

1 0.80 0.79 0.79 0.78 0.76 0.71 0.49

2

Construct Alpha

0.85

0.72 0.70

0.10


Survey Sections 1, 2 and 3 combined: 1. Overall Cronbach’s alpha (for all variables included in Sections 1, 2 and 3) was relatively high. Number of Cases 130



2. Factor analysis: Tables 13 and 14 represent the results of the factor analysis conducted for Survey Sections 1, 2 and 3. Six (6) constructs were identified, which explained about 66% of the total variance among all six (6) variables. Most of the constructs had relatively high alpha values (ranging from 0.77 to 0.95) with the exception of the last two (0.10 and 0.37).

104

Eliminated variables included Q3.09, “Project was well planned (related to people, available technology, scheduling, etc)”, Q4.04, “Customers/users did not have realistic expectations

regarding

system

functionality”,

Q4.07,

“There

was

adequate

communication between the project manager/team and customer/users” and Q4.08, “Customers/users made adequate time available for requirements gathering”. 3. Cronbach’s alpha for each identified construct from factor analysis (Step 2): The resulting Cronbach’s alpha for each of constructs are shown in Table 14. 4. The constructs were labeled as follows: •

Component 1 label: “Requirements agreement and understanding”

•

Component 2 label: “Project manager and plan”

•

Component 3 label: “Sponsor”

•

Component 4 label: “Sponsor and customer participation”

•

Component 5 label: “Size of project”

•

Component 6 label: “Project scope and meeting requirements”

Overall, there was acceptably high levels of internal consistency within the three major sections of the survey instrument (Sponsor/Management Support and Participation, Customer/Users Support and Participation, and Requirements Management). This was suggested through relatively high Cronbach’s alpha values and factor analysis (Section 8.2: Results of Analytical Methods).

Table 13: Total Variance Explained (Survey Sections 1, 2 and 3)

Component 1 2 3 4

Initial Eigenvalues % of Total Variance Cum. % 6.80 30.91 30.91 2.44 11.08 41.99 1.87 8.50 50.48 1.22 5.56 56.04

Rotation Sums of Squared Loadings % of Total Variance Cum. % 3.91 17.78 17.78 2.85 12.97 30.75 2.76 12.55 43.29 2.44 11.08 54.37

Table 13: Total Variance Explained (Survey Sections 1, 2 and 3) (continued) 5 6

1.16 1.08

5.28 4.91

61.32 66.23

1.39 1.22

6.30 5.55

60.68 66.23

105

7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

0.94 0.92 0.75 0.64 0.61 0.58 0.52 0.40 0.39 0.36 0.31 0.30 0.26 0.20 0.17 0.08

4.28 4.18 3.39 2.89 2.77 2.62 2.38 1.83 1.79 1.63 1.41 1.34 1.20 0.89 0.77 0.38

70.51 74.69 78.08 80.97 83.74 86.36 88.74 90.58 92.37 94.00 95.41 96.76 97.95 98.84 99.62 100.00

Table 14: Rotated Component Matrix (Survey Sections 1, 2 and 3) Component Variable Q5.04 Q5.07 Q5.03 Q5.05 Q5.02 Q5.08r Q3.07 Q3.04 Q3.06 Q3.05 Q3.08 Q3.03 Q3.02 Q3.01 Q4.02 Q4.03 Q4.06 Q4.01 Q4.05r Q5.06r Q5.01n Q5.09

Question Requirements accepted Team understood customer wants Requirements clear and complete Well-defined deliverables Agreement on requirements Requirements were unstable/volatile Project manager supported team Had project manager Project manager knowledgeable Project manager throughout Had project plan Sponsor comitted Sponsor throughout Had sponsor Participation was high Stayed throughout project Customer provided feedback High level of confidence Problems due to number of customers Size of project negative impact Functional scope change Requirements were met

1 0.81 0.77 0.76 0.73 0.64 0.62

2

3

4

5

6

Construct Alpha

0.87

0.77 0.77 0.76 0.66 0.55

0.79

0.91 0.89 0.86

0.95 0.75 0.74 0.72 0.47

0.77 0.87 0.48

0.37 0.72 0.59

0.10


106

8.2.2

Bivariate Correlation Analysis

Variables included in the model are discrete and measured on a five-point (interval) scale. The appropriate test of correlation between two ordinal variables is Spearman’s test [Kiess 1996]. The following steps were used to evaluate the association between independent and dependent variables within the causal model: 1. Referring to the Bayesian Network (as discussed in Section 5.1: Theoretical Model and Section 5.2: Empirical Research Model), each set of independent/dependent variables (identified by arrows originating at, and pointing to, respectively) were entered into the SPSS bivariate correlation analysis through an SPSS syntax script. Relevant SPSS parameters for conducting bivariate correlations included the following: •

Correlation coefficients: Spearman’s.

•

Test of significance:

•

Missing values:

Two-tailed. Exclude cases pairwise.

The presence of significant correlations between pairs of independent/dependent variables would provide support for the Bayesian model (as detailed by the research questions from Section 5.3). It should be noted that the bivariate analysis only considers the explanatory strength of each independent variable in isolation (i.e., any effects of any inter-correlations between any other independent variable is not included). Partial correlation analysis can isolate and account for the impact of simultaneous independent variables. However, partial correlation analysis assumes symmetric variables that are measured on at least an interval measurement scale, which is not the case in this study. The results of each bivariate analysis are shown below, arranged by research question.

Results of bivariate correlation analysis for Research Question 1: All correlations proposed in Research Question 1 were found to be significant. The dependent variable was Q4.01, “Overall, customer/users had a high level of confidence in the project

107

manager/development team”. Relative strengths of the correlation coefficients were either “low” or “moderate”. See Table 15.

Table 15: Bivariate Correlations (Spearman’s) For Research Question 1 (Is there a significant positive correlation between having a committed sponsor/champion, users that made adequate time available for requirements gathering and well-defined software deliverables, and a high level of confidence by the customer/users in the development team?) Correlated With: Q4.01: High level of user confidence Relative Strength Independent Variables Coefficient of Correlation* Significance Q3.03: Sponsor was committed 0.20 Low 0.02 Q4.08: Users made adequate time 0.45 Moderate 0.00 Q5.05: Well-defined software deliverables 0.43 Moderate 0.00

*

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 2: All correlations proposed in Research Question 2 were found to be significant. The dependent variable was Q4.02, “The level of customer/users participation during the development process was high”. Relative strengths of the correlation coefficients were either “low” or “moderate”. See Table 16.

Table 16: Bivariate Correlations (Spearman’s) For Research Question 2 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, having a committed sponsor/champion and users that made adequate time available for requirements gathering, and a high level of customer/user participation?)

Independent Variables Q3.02: Sponsor throughout project Q3.03: Sponsor was committed Q4.08: Users made adequate time

*

[Guilford 1956]

Correlated With: Q4.02: High customer/user participation Relative Strength Coefficient of Correlation Significance 0.38 Low 0.00 0.41 Moderate 0.00 0.58 Moderate 0.00

108

Results of bivariate correlation analysis for Research Question 3: All correlations proposed in Research Question 3 were found to be significant. The dependent variable was Q4.03, “Participating customers/users stayed throughout the project”. Relative strengths of the correlation coefficients were either “low” or “moderate”. See Table 17.

Table 17: Bivariate Correlations (Spearman’s) For Research Question 3 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project and a high level of customer/user participation, and participating customer/users who stayed throughout the project?)

Independent Variables Q3.02: Sponsor throughout project Q4.02: High level of user participation

*

Correlated With: Q4.03: Participating customer/users stayed Relative Strength Coefficient of Correlation Significance 0.33 Low 0.00 0.59 Moderate 0.00

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 4: All correlations proposed in Research Question 4 were found to be significant. The dependent variable was Q4.08, “Customers/users made adequate time available for requirements gathering”. Relative strength of the correlation coefficient was “low”. See Table 18.

109

Table 18: Bivariate Correlations (Spearman’s) For Research Question 4: Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, and users that made adequate time available for requirements gathering?

Independent Variables Q3.02: Sponsor throughout project

*

Correlated With: Q4.08: Users made adequate time Relative Strength Coefficient of Correlation Significance 0.31 Low 0.00

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 5: All correlations proposed in Research Question 5 were found to be significant. The dependent variable was Q5.02, “Agreement on requirements was reached between customer/users and the development team”. Relative strengths of the correlation coefficients were “moderate”. See Table 19.

Table 19: Bivariate Correlations (Spearman’s) For Research Question 5 (Is there a significant positive correlation between a high level of customer/user participation, users that made adequate time available for requirements gathering and clear/complete requirements (welldefined project scope), and agreement on requirements between customer/users and practitioners?)

Independent Variables Q4.02: High level of user participation Q4.08: Users made adequate time Q5.03: Clear & complete requirements

*

Correlated With: Q5.02: Agreement on requirements Relative Strength Coefficient of Correlation Significance 0.43 Moderate 0.00 0.48 Moderate 0.00 0.59 Moderate 0.00

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 6: All correlations proposed in Research Question 6 were found to be significant. The dependent variable was Q5.03, “Requirements were clear and complete (scope of project’s functionality was well-defined). Relative strengths of the correlation coefficients were either “low” or “moderate”. See Table 20.

110

Table 20: Bivariate Correlations (Spearman’s) For Research Question 6 (Is there a significant positive correlation between a high level of customer/user participation and users that made adequate time available for requirements gathering, and clear/complete requirements (well-defined project scope)?)

Independent Variables Q4.02: High level of user participation Q4.08: Users made adequate time

*

Correlated With: Q5.03: Clear & complete requirements Relative Strength Coefficient of Correlation Significance 0.33 Low 0.00 0.47 Moderate 0.00

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 7: All correlations proposed in Research Question 7 were found to be significant. The dependent variable was Q5.05, “Requirements gathering resulted in well-defined software deliverables (forms, reports, utilities, etc)”. Relative strengths of the correlation coefficients were “slight”, “low” or “substantial”. See Table 21.

Table 21: Bivariate Correlations (Spearman’s) For Research Question 7 (Is there a significant positive correlation between a high level of customer/user participation, changes in the functional scope (negative correlation) and clear/complete requirements (well-defined project scope), and well-defined software deliverables?)

Independent Variables Q4.02: High level of user participation Q5.01: Functional scope changed Q5.03: Clear & complete requirements

*

Correlated With: Q5.05: Well-define software deliverables Relative Strength Coefficient of Correlation Significance 0.31 Low 0.00 -0.18 Slight 0.02 0.64 Substantial 0.00

[Guilford 1956]

Results of bivariate correlation analysis for Research Question 8: All correlations proposed in Research Question 8 were found to be significant. The dependent variable was Q6.01, which is practitioners’ overall perception of project success. Relative

111

strengths of the correlation coefficients were either “low” or “moderate”. See Table 22.

Table 22: Bivariate Correlations (Spearman’s) For Research Question 8 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, a high level of confidence by the customer/users in the development team, participating customer/users who stayed throughout the project, agreement on requirements between customer/users and practitioners, well-defined software deliverables, and practitioners’ overall perception of project success?)

Independent Variables Q3.02: Sponsor throughout project Q4.01: High level of confidence in team Q4.03: Participating customer/users stayed Q5.02: Agreement on requirements Q5.05: Well-defined software deliverables

*

Correlated With: Q6.01: Overall project success Relative Strength Coefficient of Correlation Significance 0.42 Moderate 0.00 0.42 Moderate 0.00 0.27 Low 0.00 0.53 Moderate 0.00 0.50 Moderate 0.00

[Guilford 1956]

A bivariate correlation analysis was run in order to examine the relationship between each of the five ‘success’ variables (sense of achievement, did a good job, there was a project plan, project was well planned and requirements were accepted by team as realistic/achievable) and practitioners’ overall perception of project success. These five variables represent the individual components of our working definition of project success from the perspective of software developers. Any significant correlations are an indication that respondent’s read, understood and followed the directions for variable Q6.01, which instructed them to only consider the criteria mentioned above when considering how successful they judged their particular project. See Table 23 for the results of correlation analysis, which did result in significant correlations between each component of practitioners’ overall perception of project success (Q2.14, Q2.15, Q3.08, Q3.09 and 5.04) and practitioners’ overall perception of project success (Q6.01).

112

Table 23: Bivariate Correlation (Spearman’s) For Practitioners’ Success Variables Variable Q2.14 Sense of achievement Q2.15 Q3.08 Q3.09 Q5.04

Parameter Correlation Coefficient Sig. (2-tailed) Did a good job Correlation Coefficient Sig. (2-tailed) Had Project plan Correlation Coefficient Sig. (2-tailed) Project was well-planned Correlation Coefficient Sig. (2-tailed) Requirements accepted Correlation Coefficient Sig. (2-tailed)

Q6.01 0.52 0.00 0.54 0.00 0.29 0.00 0.42 0.00 0.50 0.00

A final bivariate correlation analysis examined the relationship between practitioners’ perception of project success (expressed overall by Q6.01 and by individual success variables, Q2.14, Q2.15, Q3.08, Q3.09 and Q5.04, which were previously identified and tested against Q6.01 in Table 23 above) and the organizational/managerial perspective of project success (Q2.16, “Project was completed on time”; Q2.17, “Project was completed within budget”; Q5.09, “Project met requirements”). This analysis was done in order to offer exploratory evidence that since development projects that meet practitioners’ perception of success are also at least partially related to practitioners’ level of motivation and, in turn, productivity (see Section 2.1: Project Stakeholders), such projects are also more likely to meet organizational/managerial perception of success (i.e., project was completed on time and within budget, and met customer/user requirements). As shown by the significant correlations in Table 24, there is evidence of a relationship between practitioners’ perception of success (and presumably motivated, productive practitioners) and the organization/management’s perception. It should be noted, however, that any such preliminary conclusions must be read with caution because (1) practitioners were the source of the data (i.e., whether the project was completed on time, within budget, etc.), as opposed to a manager, customer and/or end-user who might be better informed about such matters, and (2) budgets and estimates of time can be padded and otherwise manipulated.

113

Table 24: Bivariate Correlation (Spearman’s) For Practitioners’ and Organizational Success Variables

Practitioners' Success Q2.14 Sense of achievement Q2.15 Q3.08 Q3.09 Q5.04 Q6.01

8.2.3

Parameter Correlation Coefficient Sig. (2-tailed) Did a good job Correlation Coefficient Sig. (2-tailed) Had project plan Correlation Coefficient Sig. (2-tailed) Project was well-planned Correlation Coefficient Sig. (2-tailed) Requirements accepted Correlation Coefficient Sig. (2-tailed) Overall success rating Correlation Coefficient Sig. (2-tailed)

Organizational Success Q2.17 Q2.16 Completed Q5.09 Completed within Requirements on time budget weree met 0.34 0.40 0.22 0.00 0.00 0.00 0.34 0.38 0.21 0.00 0.00 0.01 0.15 0.06 0.21 0.07 0.50 0.01 0.32 0.31 0.37 0.00 0.00 0.00 0.30 0.21 0.37 0.00 0.01 0.00 0.45 0.41 0.44 0.00 0.00 0.00

Ordinal Regression Analysis

There are a total of eight (8) independent/dependent relationships that comprise the causal relationships within the proposed Bayesian Network (see Table 25). The results of this analysis are intended to provide some insight into the explanatory power of each of the eight causal relationships within the overall model. The significance of each parameter and R-square is included in the analysis. SPSS provided three R-square calculations, which in order of least conservative to most conservative was Nagelkerke, Cox and Snell, and McFadden. This study uses Nagelkerke because it has been reported to be the best representation of R-square [Maddala 1983]. The formula for Nagelkerke’s R-square is as follows:

where w=number of observations

114

Table 25: Causal Relationships In Proposed Model Dependent Variable Q4.01 Q4.02 Q4.03 Q4.08 Q5.02 Q5.03 Q5.05 Q6.01

*

Question Customers had high confidence in PM/team. Level of customer/user participation was high. Participating customer/users stayed throughout. User made adequate time for requirements. Agreement on requirements was reached. Requirements were clear and complete. Requirements resulted in well-defined deliverables. Practitioner’s overall perception of project success.

Original Independent Variables* Q3.03, Q4.08, Q5.05 Q3.02, Q3.03, Q4.08 Q3.02, Q4.02 Q3.02 Q4.02, Q4.08, Q5.03 Q4.02, Q4.08 Q4.02, Q5.01, Q5.03 Q4.01, Q5.05, Q5.02, Q4.03, 3.02

Each independent variable was recoded into 4 dummy variables (see below). Descriptions of dependent variables included the following: Q3.02, “Sponsor throughout project” Q3.03, “Sponsor was committed” Q4.01, “High level of confidence of users in project manager/team” Q4.02, “Level of customer/user participation was high” Q4.03, “Participating customer/users stayed throughout” Q4.08, “Users made adequate time available for requirements gathering” Q5.01, “Functional scope changed” Q5.02, “Agreement on requirements reached between users and team” Q5.03, “Requirements were clear and complete” Q5.05, “Requirements gathering resulted in well-defined software deliverables” Q6.01, “Practitioner’s overall perception of project success”

The following steps will be used to conduct the ordinal regressions and calculate associated Rsquare values: 1. In order to ensure a consistent scale across all variables, dummy variables were created for every independent variable (see Table 25) as follows. Each independent variable is based on a five-point scale. As a result, independent variables in ordinal regression analysis need to be converted to dummy variables so that an interval scale is not assumed. Each of the independent variables were converted to four dummy variables, each coded as 1 (yes) or 0 (no), which corresponded to original responses of 1, 2, 3 or 4. The value corresponding to five (5) is reflected in the resulting constant term. 2. The appropriate dependent variable was entered into the SPSS ordinal regression analysis. Default SPSS parameters were used when running the analysis. (Unlike continuous variables, non-significant parameters cannot be attributed to any multicollinarity among the variables because the values of the binomial variables are mutually exclusive.) 3. Associated sets of the four dummy variables for each independent variable(s) were entered into the SPSS ordinal regression analysis. 4. Resulting significant parameter estimates and R-square values were noted. Attempts were made to collapse variables where appropriate, and significant parameter

115

estimates and R-square values were noted. The choice of the final regression analysis was based on which analysis resulted in the highest number of significant parameter estimates and highest R-square value. In addition, some of the regression analsyses needed further adjustment to adjust for an error generated by SPSS, which advised that due to a quasi-separation of the data (related to empty cells resulting from the combination of independent and dependent variables), some of the analyses was producing parameter estimates that would “tend to infinity”. As a result, the validty of the results of the analysis were “uncertain”. A final regression analysis run in order to produce an overall R-square, without regard to a predetermined casual chain of project variables, which was subsequently used to test the model’s goodness-of-fit. All variables in the Bayesian Belief Network were regressed on the ultimate dependent variable, Q6.01.

The results of each ordinal regression analysis are shown below, arranged by research question.

116

Results of ordinal regression analysis for Research Question 1: Table 26 shows the results of the ordinal regression analysis performed on the dependent variable, Q4.01, “High level of confidence in PM/team” (includes only significant parameter estimates). The resulting R-square value was 0.28. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. This intuitively makes sense because we would expect a stronger agreement with each of the dependent variables (each coded so a higher response indicates stronger agreement) to be associated with stronger agreement of a high level of confidence in the project manager/development team. Significant independent variables in the regression analysis of a high level of confidence in project manager/team included users making adequate time for requirements gathering and requirements resulting in well-defined software deliverables.

Table 26: Ordinal Regression Results For Research Question 1 (Is there a significant positive correlation between having a committed sponsor/champion, users that made adequate time available for requirements gathering and well-defined software deliverables, and a high level of confidence by the customer/users in the development team?) Dummy Variable* Thresholds: Q4.01=1 Q4.01=2 Q4.01=3 Q4.01=4 Locations: Q3.03=1 Q3.03=2,3,4 Q4.08=1 Q4.08=2,3,4 Q5.05=1 Q5.05=2,3,4

*

Original Question

Estimate Std. Error Wald Sig.

High level of confidence in PM/team High level of confidence in PM/team High level of confidence in PM/team High level of confidence in PM/team

-7.65 -6.89 -3.66 -1.47

1.13 0.90 0.52 0.44

46.28 59.05 49.45 11.20

0.00 0.00 0.00 0.00

Sponsor was committed Sponsor was committed User made adequate time for reqt’s User made adequate time for reqt’s Reqt’s resulted in well-defined deliverables Reqt’s resulted in well-defined deliverables

-3.04 -0.39 -2.81 -1.32 -2.46 -0.91

2.13 2.04 0.15 0.33 1.37 0.24 0.08 13.17 0.00 0.41 10.29 0.00 1.06 5.43 0.02 0.40 5.31 0.02

All parameters had 1 degree of freedom There were 35 (53.8%) empty cells (combinations of levels of dependent variable and independent variables)

117

Results of ordinal regression analysis for Research Question 2: Table 27 shows the results of the ordinal regression analysis performed on the dependent variable, Q4.02, “Level of customer/user participation was high” (includes only significant parameter estimates). The resulting R-square value was 0.46. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. (Q3.03, “Sponsor was committed”, was an exception.) Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that there was a high level of customer/user participation. Significant independent variables in the regression analysis of a high level of customer/user participation included a committed sponsor and users making adequate time for requirements gathering.

Table 27: Ordinal Regression Results For Research Question 2 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, having a committed sponsor/champion and users that made adequate time available for requirements gathering, and a high level of customer/user participation?) Dummy Variable* Thresholds: Q4.02=1 Q4.02=2 Q4.02=3 Q4.02=4 Locations: Q3.02=1,2 Q3.02=3,4 Q3.03=1,2 Q3.03=3,4 Q4.08=1 Q4.08=2 Q4.08=3 Q4.08=4

*

Original Question

Estimate

Standard Error Wald Sig.

Level of customer/user participation was high Level of customer/user participation was high Level of customer/user participation was high Level of customer/user participation was high

-7.13 -5.00 -3.51 -1.51

0.79 0.57 0.49 0.41

82.45 76.26 51.10 13.85

0.00 0.00 0.00 0.00

Sponsor throughout project Sponsor throughout project Sponsor was committed Sponsor was committed User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering

-1.07 0.26 -0.92 -1.43 -4.93 -3.17 -2.41 -1.41

1.37 0.61 0.43 0.69 0.15 0.70 1.20 0.59 0.44 0.70 4.23 0.04 0.80 37.69 0.00 0.63 25.03 0.00 0.53 20.43 0.00 0.46 9.38 0.00


118

Results of ordinal regression analysis for Research Question 3: Table 28 shows the results of the ordinal regression analysis performed on the dependent variable, Q4.03, “Participating customer/users stayed throughout” (includes only significant parameter estimates). The resulting R-square value was 0.43. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that participating customer/users stayed throughout the development process. Significant independent variables in the regression analysis of participating customer/users staying throughout the development process included a sponsor who stayed throughout the development process and a high level of customer/user participation.

Table 28: Ordinal Regression Results For Research Question 3 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project and a high level of customer/user participation, and participating customer/users who stayed throughout the project?) Dummy Variable* Thresholds: Q4.03=1 Q4.03=2 Q4.03=3 Q4.03=4 Locations: Q3.02=1,2,3 Q3.02=4 Q4.02=1,2 Q4.02=3 Q4.02=4

*

Original Question

Estimate


Participating customer/users stayed Participating customer/users stayed Participating customer/users stayed Participating customer/users stayed

-7.56 -5.12 -3.88 -1.78

0.87 0.54 0.47 0.37

74.86 89.45 68.16 23.36

0.00 0.00 0.00 0.00

Sponsor throughout project Sponsor throughout project Level of customer/user participation was high Level of customer/user participation was high Level of customer/user participation was high

-0.87 -0.37 -4.42 -2.94 -2.02

0.43 4.07 0.04 0.42 0.78 0.38 0.64 48.14 0.00 0.54 29.16 0.00 0.45 20.58 0.00


119

Results of ordinal regression analysis for Research Question 4: Table 29 shows the results of the ordinal regression analysis performed on the dependent variable, Q4.08, “Users made adequate time for requirements gathering” (includes only significant parameter estimates). The resulting R-square value was 0.15. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that users made adequate time for requirements gathering. Significant independent variables in the regression analysis of users making adequate time for requirements gathering included a sponsor who stayed throughout the development process.

Table 29: Ordinal Regression Results For Research Question 4 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, and users that made adequate time available for requirements gathering?) Dummy Variable* Thresholds: Q4.08=1 Q4.08=2 Q4.08=3 Q4.08=4 Locations: Q3.02=1 Q3.02=2 Q3.02=3,4

*

Original Question

Estimate


User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering

-3.38 -2.08 -0.99 0.64

0.41 68.42 0.00 0.28 54.97 0.00 0.23 18.59 0.00 0.22 8.52 0.00

Sponsor throughout project Sponsor throughout project Sponsor throughout project

-4.58 -1.33 -0.80

1.24 13.67 0.00 0.67 3.92 0.05 0.32 6.33 0.01


120

Results of ordinal regression analysis for Research Question 5: Table 30 shows the results of the ordinal regression analysis performed on the dependent variable, Q5.02, “Agreement on requirements was reached between customer/users” (includes only significant parameter estimates). The resulting R-square value was 0.49. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. (Q4.08, “Users made adequate time for requirements gathering”, was an exception.) Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that agreement on requirements was reached between customer/users and the development team. Significant independent variables in the regression analysis of reaching an agreement on requirements included a high level of customer/user participation, users making adequate time for requirements gathering, and requirements were clear and complete.

Table 30: Ordinal Regression Results For Research Question 5 (Is there a significant positive correlation between a high level of customer/user participation, users that made adequate time available for requirements gathering and clear/complete requirements (well-defined project scope), and agreement on requirements between customer/users and practitioners?) Dummy Variable* Thresholds: Q5.02=2 Q5.02=3 Q5.02=4 Locations: Q4.02=1,2,3 Q4.02=4 Q4.08=1,2,3 Q4.08=4 Q5.03=1,2,3 Q5.03=4

*

Original Question

Estimate


Agreement on requirements reached Agreement on requirements reached Agreement on requirements reached

-7.79 -5.55 -2.45

0.82 91.46 0.00 0.69 64.12 0.00 0.56 19.51 0.00

Level of customer/user participation was high Level of customer/user participation was high User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering Requirements were clear and complete Requirements were clear and complete

-1.17 -0.19 -1.32 -1.18 -3.65 -1.96

0.50 5.50 0.02 0.45 0.18 0.67 0.56 5.40 0.02 0.51 5.35 0.02 0.62 34.88 0.00 0.54 13.42 0.00

All parameters had 1 degree of freedom There were 43 (44.8%) empty cells (combinations of levels of dependent variable and independent variables) Results of ordinal regression analysis for Research Question 6:

121

Table 31 shows the results of the ordinal regression analysis performed on the dependent variable, Q5.03, “Requirements were clear and complete” (includes only significant parameter estimates). The resulting R-square value was 0.28. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that requirements were clear and complete. Significant indendepent variables in the regression analysis of clear and complete requirements included users making adequate time for requirements gathering.

Table 31: Ordinal Regression Results For Research Question 6 (Is there a significant positive correlation between a high level of customer/user participation and users that made adequate time available for requirements gathering, and clear/complete requirements (well-defined project scope)?) Dummy Variable* Thresholds: Q5.03=1 Q5.03=2 Q5.03=3 Q5.03=4 Locations: Q4.02=1 Q4.02=2,3,4 Q4.08=1 Q4.08=2 Q4.08=3 Q4.08=4

*

Original Question

Estimate


Requirements were clear and complete Requirements were clear and complete Requirements were clear and complete Requirements were clear and complete

-6.54 -3.86 -2.42 0.05

0.83 61.80 0.00 0.45 73.97 0.00 0.38 39.72 0.00 0.31 0.03 0.87

Level of customer/user participation was high Level of customer/user participation was high User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering

-1.42 -0.27 -3.28 -2.78 -1.50 -1.30

0.97 2.14 0.14 0.37 0.51 0.48 0.73 20.35 0.00 0.63 19.74 0.00 0.52 8.27 0.00 0.43 9.00 0.00


122

Results of ordinal regression analysis for Research Question 7: Table 32 shows the results of the ordinal regression analysis performed on the dependent variable, Q5.05, “Well-defined software deliverables” (includes only significant parameter estimates). The resulting R-square value was 0.44. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. (Q5.01, “Functional scope changed”, was an exception. However, this variable was not significant.) Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that software deliverables were well-defined. Significant independent variables in the regression analysis of well-defined software deliverables included a high level of customer/user participation, and requirements were clear and complete.

Table 32: Ordinal Regression Results For Research Question 7 (Is there a significant positive correlation between a high level of customer/user participation, changes in the functional scope (negative correlation) and clear/complete requirements (welldefined project scope), and well-defined software deliverables?) Dummy Variable* Thresholds: Q5.05=1 Q5.05=2 Q5.05=3 Q5.05=4 Locations: Q4.02=1,2,3 Q4.02=4 Q5.01=1,2,3 Q5.01=4 Q5.03=1,2 Q5.03=3,4

*

Original Question

Estimate


Well-defined software deliverables Well-defined software deliverables Well-defined software deliverables Well-defined software deliverables

-6.73 -5.47 -3.62 -0.89

0.83 65.69 0.00 0.74 55.28 0.00 0.68 28.46 0.00 0.63 2.01 0.16

Level of customer/user participation was high Level of customer/user participation was high Functional scope changed Functional scope changed Requirements were clear and complete Requirements were clear and complete

-1.05 -0.19 0.58 0.65 -4.79 -2.78

0.41 6.62 0.01 0.40 0.23 0.63 0.53 1.19 0.28 0.47 1.90 0.17 0.69 48.62 0.00 0.50 30.48 0.00


123

Results of ordinal regression analysis for Research Question 8: Table 33 shows the results of the ordinal regression analysis performed on the dependent variable, Q6.01, “Practitioners’ overall perception of project success” (includes only significant parameter estimates). The resulting R-square value was 0.32. In general, the regression estimate for each of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. (Q3.02, “Sponsor throughout the project”, was an exception, but was not significant. However, Q5.05, “Well-defined software deliverables”, was also an exception, and was significant.) Again, this intuitively makes sense because we would expect a stronger agreement with each of the dependent variables to be associated with stronger agreement that the project was considered to be a success as defined in this study. Significant independent variables in the regression analysis of practitioners’ overall perception of project succes included agreement was reached on requirements and well-defined software deliverables.

Table 33: Ordinal Regression Results For Research Question 8 (Is there a significant positive correlation between having a sponsor/champion that stayed throughout the project, a high level of confidence by the customer/users in the development team, participating customer/users who stayed throughout the project, agreement on requirements between customer/users and practitioners, well-defined software deliverables, and practitioners’ overall perception of project success?) Dummy Variable* Thresholds: Q6.01=0 Locations: Q3.02=1,2,3 Q3.02=4 Q4.01=1,2,3 Q4.01=4 Q4.03=1,2,3 Q4.03=4 Q5.02=1,2,3 Q5.02=4 Q5.05=1,2,3

Original Question Practitioners’ overall perception of project success Sponsor throughout project Sponsor throughout project High level of confidence in PM/team High level of confidence in PM/team Participating customer/users stayed Participating customer/users stayed Agreement on requirements reached Agreement on requirements reached Well-defined software deliverables

Estimate

Standard Error

Wald Sig.

1.57

0.46

11.62 0.00

0.80 -0.14 -0.68 0.41 -0.63 -0.61 -0.24 1.51 1.15

0.55 0.48 0.65 0.47 0.62 0.48 0.71 0.50 0.61

2.17 0.09 1.09 0.75 1.02 1.61 0.12 8.93 3.58

0.14 0.77 0.30 0.39 0.31 0.21 0.74 0.00 0.06

124

Table 33 (cont’d) Dummy Variable* Q5.05=4

*

Original Question Well-defined software deliverables

Estimate 1.33

Standard Error 0.50

Wald Sig. 7.01 0.01


Regression Analysis of Overall Model: For purposes of comparison, an ordinal regression analysis was run for the entire model to arrive at an ‘overall’ R-square. That is, all variables in the model were simultaneously regressed to practitioners’ overall perceptive of project success (Q6.01). The results are show in Table 34. The resulting R-square value of 0.44 was then used in a test of goodness-of-fit of the model (see Section 8.2.4: Bayesian Belief Network). The regression estimate for half of the dependent variables increased as the value of their associated independent variable increased, comparing responses to associated parameter estimates. (The following variables were exceptions: Q3.03, “Sponsor was committed”; Q4.03, “Participating customer/users stayed throughout”; Q4.08, “Users made adequate time for requirements gathering”; Q5.05, “Well-defined software deliverables”. However, only Q4.08 and Q5.05 were significant. See the next paragraph.) Significant independent variables in the overall regression analysis of practitioners’ overall perception of project success included users making adequate time for requirements gathering, agreement was reached on requirements, and well-defined software deliverables.

As mentioned in prior regression analyses, the coefficient of Q4.08, “Users made adequate time for requirements gathering”, increased in the analysis of the following dependent variables: Q4.01, “High level of confidence in PM/team”; Q4.02, “Level of customer/user participation was high”; Q5.02, “Agreement on requirements was reached between customer/users”; Q5.03, “Requirements were clear and complete”. Further, Q4.08 was significant in each of these analyses. Also as mentioned previously, Q5.05, “Well-defined software deliverables”, increased

125

in the analysis of the dependent variables Q4.01 and Q6.01, “Practitioners’ overall perception of project success”. However, Q5.05 was only significant in the analysis of Q4.01.

Table 34: Overall Ordinal Regression Results Dummy Variable* Thresholds: Q6.01=0 Locations: Q3.02=1,2,3 Q3.02=4 Q3.03=1,2,3 Q3.03=4 Q4.01=1,2,3 Q4.01=4 Q4.02=1,2,3 Q4.02=4 Q4.03=1,2,3 Q4.03=4 Q4.08=1,2,3 Q4.08=4 Q5.01=1,2,3 Q5.01=4 Q5.02=1,2,3 Q5.02=4 Q5.03=1,2,3 Q5.03=4 Q5.05=1,2,3 Q5.05=4

*

Original Question Practitioners’ overall perception of project success Sponsor throughout project Sponsor throughout project Sponsor was committed Sponsor was committed High level of confidence in PM/team High level of confidence in PM/team Level of customer/user participation was high Level of customer/user participation was high Participating customer/users stayed Participating customer/users stayed User made adequate time for reqt’s gathering User made adequate time for reqt’s gathering Agreement on requirements reached Agreement on requirements reached Agreement on requirements reached Agreement on requirements reached Requirements were clear and complete Requirements were clear and complete Well-defined software deliverables Well-defined software deliverables

Estimate

Standard Error

Wald Sig.

2.94

0.92

10.20 0.00

1.20 0.12 0.07 -0.03 -0.95 0.02 -1.16 0.06 -0.58 -1.10 2.07 1.49 0.30 0.38 -0.13 1.41 -0.66 0.78 1.48 1.28

1.20 0.83 1.18 0.89 0.73 0.55 0.87 0.69 0.79 0.62 0.77 0.66 0.75 0.67 0.83 0.57 0.86 0.71 0.77 0.61

1.00 0.02 0.00 0.00 1.70 0.00 1.78 0.01 0.54 3.09 7.15 5.11 0.16 0.32 0.02 6.15 0.60 1.20 3.69 4.32

0.32 0.89 0.95 0.97 0.19 0.97 0.18 0.93 0.46 0.08 0.01 0.02 0.69 0.57 0.88 0.01 0.44 0.27 0.05 0.04


The summary R-square of each of the eight (8) ordinal regression models is presented in Table 35.

126

Table 35: Summary of Ordinal Regression Models Dependent Variable Q4.01 Q4.02 Q4.03 Q4.08 Q5.02 Q5.03 Q5.05 Q6.01

Question Customers had high confidence. High customer/user participation. Participating customer/users stayed. User made adequate time for reqts. Agreement reached on requirements. Requirements were clear and complete. Reqts resulted in well-defined delivera. Practitioner’s perception of success.

Cox and Snell R-square 0.28 0.46 0.43 0.15 0.49 0.28 0.44 0.32

There was evidence to support the eight (8) causal relationships within the Bayesian model based on correlation and ordinal regression analysis. Significant bivariate correlations were found between each of the proposed independent and dependent variables within the model (Section 8.2.2: Bivariate Correlation Analysis). Further, there was evidence to conclude that these causal models did a better job of predicting these dependent variables than did simultaneously regressing all variables in the model on the final dependent variable, practitioners’ overall perception of project success (Q6.01). However, most of the causal relationships embedded within the Bayesian model resulted in low to moderate levels of explained variance in their respective dependent variable (Section 8.2.3: Ordinal Regression Analysis), depending on the particular calculation of R-square.

8.2.4

Bayesian Belief Network

The proposed Bayesian Network depicts some of the early and interrelated events within the software development process that must be addressed, primarily by the project manager, in order to increase the probability (and subsequently decrease the project risk) of a project being viewed as a success from the perspective of software practitioners. As discussed earlier (Chapter 3: NonTechnical Components of Software Development), the causal model was based on those models developed in Procaccino, et al [2000a and 200b]. This model expands on Procaccino, et al [2000a and 200b] through the addition of variables that the literature review suggests may have a causal

127

relationship with practitioners’ overall perception of project success, as defined by Pilot Study #2. Many additional variables were investigated (and are included in the final survey instrument). However, they were not included in the proposed model because they either had relatively weak support in the literature, were only related to the organizational/managerial perception of success and/or resulted in causal ‘dead ends’ (i.e. did not relate, either directly or indirectly, to practitioner’s overall perception of success Q6.01).

The resulting proposed Bayesian Model is shown in Figure 15. The model represents a combination of each of the causal relationships investigated in Research Questions 1 through 9 (Section 5.3: Research Questions) and depicted in Figures 6 through 13. The model provides evidence of the probability of each combination of independent/dependent relationships. These probabilities can be expressed thought the following belief measures [Agena 2002a], which equate to every combination of dependent variable given the condition (value or state; denoted by | ) of its associated independent variables. • • • • • • • • • • •

P(Q3.02) P(Q3.03) P(Q5.01) P(Q4.01|Q3.03, Q4.08, Q5.05) P(Q4.02|Q3.02, Q3.03, Q4.08) P(Q4.03|Q3.02, Q4.02) P(Q4.08|Q3.02) P(Q5.02|Q4.02, Q4.08, Q5.03) P(Q5.03|Q4.02, Q4.08) P(Q5.05|Q5.01) P(Q6.01|Q3.02, Q4.01, Q4.03, Q5.02, Q5.05)

As noted earlier (Section 8.1.4: Bayesian Belief Networks), each of the arcs (arrows) in the model originates at an independent variable (a node, shown as an oval) and end at (point to) a dependent variable (also shown as an oval node). For example, changes in the functional scope of the project (Q5.01) influences whether requirements gathering resulted in well-defined software deliverables (Q5.05). Well-defined software deliverables, in turn, influences whether customer/users have a

128

high level of confidence in the project manager/development team (Q4.01).

The following steps were used to construct and test the proposed Bayesian model: 1. Frequencies were generated for each of the three (3) variables that were solely independent variables (i.e., not dependent variables) within the overall model. The independent variables are shown with their respective frequency distributions in Table 36.

Table 36: Frequencies of Independent Variables (Q3.02, Q3.03 and Q5.01) Responses 1 2 3 4 5 Totals

Q3.02 Valid % 2.6 5.2 15.5 21.9 54.8 100.0

Q3.03 Valid % 0.6 9.0 12.2 19.9 58.3 100.0

Q5.01 Valid % 0.0 10.7 18.9 57.2 13.2 100.0

2. Frequencies were generated for each of the eight (8) causal models within the Bayesian Belief Network, each consisting of (percentage/probabilities) of every combination of independent/dependent variables. As mentioned in Section 8.1.4: Bayesian Belief Networks, all combinations of independent/dependent variables within each of the eight causal models. (The total number of cells was 18,415, including those from Step 1 above. However, due to the relatively small number of observations, most of the cells were null. Default SPSS parameters were used when generating the frequencies.) The eight models have the following as their dependent variable, respectively: Q4.01, Q4.02, Q4.03, Q4.08, Q5.02, Q5.03, Q5.05 and Q6.01. Figure 15 presents a graphical depiction of each of these relationships, including variable (survey) numbers. The model’s ultimate dependent variable is practitioners’ overall perception of project success (Q6.01), which is located in the lower right-hand corner of Figure 15.

129

Figure 15: Proposed Bayesian Belief Network With Bivariate Correlation Coefficients (and significance levels)

3. Frequencies were entered into the Hugin Expert System (www.hugin.com). 4. A goodness-of-fit test was conducted to evaluate the relative appropriateness of the model. The test was conducted by comparing the generalized squared multiple correlation to the overall R-square described in Section 8.2.3: Ordinal Regression Analysis [Schumacker and Lomax 1996; Evanco, et al 2002]. Eight ordinal regression analysis were run, resulting in eight R-square values (each R-square in the equation is identified by a subscript of the associated dependent variable). The generalized squared multiple correlation utilizes each of these R-square values (see below) and compares the result to the R-square obtained from simultaneously regressing all variables on practitioners’ overall perception of project success (Q6.01; see end of Section 8.2.3: Ordinal Regression Analysis). When the generalized squared multiple correlation is larger than the overall R-square, then we conclude that the model is reasonable (i.e., it is a better representation of the data than assuming no causal connections among the variables included in the model). The generalized squared multiple correlation was calculated as follows: = 1 – (1-R24.01) * (1-R24.02) * (1-R24.03) * (1-R24.08) * (1-R25.02) * (1-R25.03) * (1-R25.05) * (1-R26.01)

Substituting the R-square values from Table 35 into the above equation yields the following calculation of the generalized squared multiple correlation.

130

= 1 – (1-0.28) * (1-0.46) * (1-0.43) * (1-0.15)*(1-0.49) * (1-0.28) * (1-0.44) * (1-0.32) = 0.97

The generalized squared multiple correlation (0.97) is larger than the overall Rsquare (0.44; from end of Section 8.2.3: Ordinal Regression Analysis). This infers that the model has more predictive power than simply regressing all variables to practitioners’ overall perception of project success (Q6.01). The overall propagated probabilities from the model for practitioners’ overall perception of project success are shown in Table 37. Note should be made of the relatively low counts (N) associated with responses 1, 2 and 3.

Table 37: Overall Propagated Probabilities For Q6.01, “Practitioners’ Overall Perception of Success” Response 1 (disagree) 2 3 4 5 (agree) Totals

*

Probability 11.62% 11.64% 13.25% 29.68% 33.81% 100.00%

Count (N*) 1 3 11 82 65 162

Frequency Percent 0.62% 1.85% 6.79% 50.62% 40.12% 100.00%

6 missing values

The findings that were derived from the Bayesian model are presented below as follows. First, the overall probabilities of each variable in the model equating to each of the five possible responses according to the BBN are presented. An analysis was then conducted to determine the composition of a ‘critical path’ (i.e., the causal series of variables that exhibit the greatest impact on project success). The next finding presents the investigation to determine which of the model’s variables, if any, exhibited the greatest impact on project success, as measured by the probability of variable, Q6.01, equating to either 4 or 5 on a five-point Likert scale.

1. Overall Propogated Probabilities of Each Variable In Model: Table 38 summaries the overall Bayesian probabilities from the proposed model. The

131

probabilities of each variable included in the model equaling 1 through 5 are listed in the appropriate

column.

For

example,

given

all

of

the

empirical

evidence

and

the

independent/dependent relationships, the overall probability of Q3.02 equaling 5 (“agree”) was 54.80%.

Table 38: Original Probability Propagation Variable Q3.02 Q3.03 Q4.01 Q4.02 Q4.03 Q4.08 Q5.01 Q5.02 Q5.03 Q5.05 Q6.01

1 2.60 0.60 4.10 9.22 1.85 7.20 0.00 4.78 2.31 4.06 11.62

2 5.20 9.00 4.39 12.30 6.72 11.08 10.70 7.63 13.52 8.85 11.64

Response 3 4 15.50 21.90 12.20 19.90 16.83 37.03 14.48 29.45 9.35 30.57 18.40 36.27 18.90 57.20 20.50 35.62 21.06 36.67 17.63 43.47 13.25 29.68

5 54.80 58.30 37.65 34.55 51.51 27.05 13.20 31.47 26.44 25.99 33.81

Total 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00

Figure 16 shows the overall result of propagating the data through the Hugin System, which displays the results shown in Table 38 through monitor windows.

132

Figure 16: Overall Propagated Probabilities Through Hugin’s Monitor Windows

2. Identification of Critical Path: It is valuable to investigate which specific causal chain within the Bayesian Belief Network demonstrated the greatest impact on practitioners’ overall perception of project success (Q6.01, the ultimate dependent variable). This is accomplished by determining which variable had the greatest impact on Q6.01 by tracing backwards through the Bayesian network from Q6.01 (the ultimate dependent variable). The candidate independent variables of Q6.01 included Q3.02, Q4.01, Q4.03, Q5.02 and Q5.05. This finding was made through isolating the effect of each independent variable on practitioners’ overall perception of project success (Q6.01). Specifically, the value of each independent variable was held to a particular value (1, 2, 3, 4 or 5) for all cases. The effect of this constraint on the final dependent variable, Q6.01, was then recorded. The focus was on results associated with independent variables held to 4 and 5 (agree) and the dependent variable, Q6.01, equaling 4 or 5 (‘successful’ projects). The value of 1 or 2 for independent and dependent variables was not considered due to a relatively low number of responses in independent and dependent (Q6.01) variables. Further, the value “3” was not directly considered

133

due to a lack of direction (“agree” or “disagree”) in the respondent’s response regarding both independent and dependent variables. However, for completeness, all responses are detailed in Appendix R. The next step was to determine which independent variable had the greatest impact on that ‘upstream’ variable, and so on.

In order to provide a more detailed explanation of this analysis, consider the analysis to determine the variable that has the largest impact on Q6.01. As mentioned previously, variable Q3.02, “There was an sponsor throughout this project”, was a candidate independent variable of Q6.01. The steps in the analysis of this variable included the following: 1. Q3.02 was forced to a value (state) of 4 for all of the cases (a hypothetical constraint). 2. All of the probabilities within the model were then re-propagated given this new ‘evidence’ and the revised probability of practitioners’ overall perception of project success (Q6.01) equating to a value of 4 or 5 was recorded (41.18% and 17.22%, respectively, given Q3.02=4). 3. The overall probability of Q6.01 equaling 4 or 5 when Q3.02 equaled 4 was computed by adding the probabilities from Step 2 (41.18%+17.22%=58.40%). This could be stated as follows: = P(Q6.01=4|Q3.02=4) + P(Q6.01=5|Q3.02=4)

4. The BBN was then re-initialized to its original state (probabilities) and Q3.02 was then forced to a value (state) of 5 (agree) for all of the cases (again, a hypothetical constraint). 5. All of the probabilities within the BBN were then re-propagated given this new ‘evidence’ and the revised probabilities of Q6.01 taking on the value of either 4 or 5 were recorded (27.34% and 46.05%, respectively). 6. The overall probability of Q6.01 equaling 4 or 5 when Q3.02 equaled 5 was computed by adding the probabilities from Step 4 (27.34%+46.05%=73.39%). This could be stated as follows: = P(Q6.01=4|Q3.02=5) + P(Q6.01=5|Q3.02=5)

7. The average of the probability of Q6.01 equaling 4 or 5 when Q3.02 equaled 4 or 5 was computed by averaging the probabilities from Step 5 and 6 (average of 58.40% and 73.39%=65.90%) in order to arrive at a measure of overall impact of Q3.02 on Q6.01.

Similar analysis was then performed on each of the other independent variables within the model.

134

Table 39 summaries the results of this analysis of which variable had the greatest impact on Q6.01, as measured by the average percentage of projects that were considered to be a success (Q6.01=4 or 5). These findings are also graphically depicted in Figure 17. No single variable exhibited an overwhelming impact on a project being considered a success. However, given the data collected, Q5.02, “Agreement on requirements reached between customer/users” had the largest impact on Q6.01 (74.13% average probability that Q6.01=4 or 5).

Table 39: Impact Summary For Q6.01, “Practitioners’ Overall Perception of Success” Variable Q3.02 Q4.01 Q4.03 Q5.02 Q5.05

Question Sponsor throughout project: =P(Q6.01=4|Q3.02=4)+P(Q6.01=5|Q3.02=4)=41.18%+17.22%=58.40% =P(Q6.01=4|Q3.02=5)+P(Q6.01=5|Q3.02=5)=27.34%+46.05%=73.39% Customer/users had high level of confidence in PM/development team: =P(Q6.01=4|Q4.01=4)+P(Q6.01=5|Q4.01=4)=35.54%+33.44%=68.98% =P(Q6.01=4|Q4.01=5)+P(Q6.01=5|Q4.01=5)=27.13%+42.76%=69.89% Participating customer/users stayed throughout project: =P(Q6.01=4|Q4.03=4)+P(Q6.01=5|Q4.03=4)=37.28%+21.80%=59.08% =P(Q6.01=4|Q4.03=5)+P(Q6.01=5|Q4.03=5)=27.53%+45.61%=73.14% Agreement on requirements reached between customer/users: =P(Q6.01=4|Q5.02=4)+P(Q6.01=5|Q5.02=4)=46.18%+24.96%=71.14% =P(Q6.01=4|Q5.02=5)+P(Q6.01=5|Q5.02=5)=17.88%+59.24%=77.12% Requirements gathering resulted in well-defined software deliverables: =P(Q6.01=4|Q5.05=4)+P(Q6.01=5|Q5.05=4)=36.32%+34.74%=71.06% =P(Q6.01=4|Q5.05=5)+P(Q6.01=5|Q5.05=5)=20.70%+46.10%=66.80%

Average Percent 65.90% 69.44% 66.11% 74.13% 68.93%

135

100

Percentage of Q6.01

90 80 70 60

73.4

73.1

69.0 69.9

71.1

77.1

71.1

66.8

59.1

58.4

50 40 30 20 10 Q3.02

Q4.01

Q4.03 Q6.01="4"

Q5.02

Q5.05

Q6.01="5"

Figure 17: Percentage of Q6.01=4 or 5 When All Cases of Each Independent Variable=4 or 5.

Since Q5.02 had the largest impact on Q6.01, the next step was to trace which variable is the largest influencer of Q5.02. Candidate independent variables included Q4.02, “High level of customer/user participation”; Q4.08, “Users made adequate time for requirements gathering”; Q5.03. “Requirements were clear and complete” (see Figure 15). Table 40 summaries the results of this analysis of which variable had the greatest impact on Q5.02, as measured by the average percentage of projects that were considered to have agreement between customer/users and the development team on requirements (Q5.02=4 or 5). These findings are also graphically depicted in Figures 18. No single variable exhibited an overwhelming impact on agreement between customer/users and the development team on requirements. However, given the data collected, Q4.02, “High level of customer/user participation” had the largest impact on Q5.02 (78.77% average probability that Q5.02=4 or 5).

Table 40: Impact Summary For Q5.02, “Agreement on requirements reached” Average

136

Variable Q4.02

Question High level of customer/user participation: =P(Q5.02=4|Q4.02=4)+P(Q5.02=5|Q4.02=4)=58.56%+21.49%=80.05% =P(Q5.02=4|Q4.02=5)+P(Q5.02=5|Q4.02=5)=21.84%+55.64%=77.48% Users made adequate time for requirements gathering: =P(Q5.02=4|Q4.08=4)+P(Q5.02=5|Q4.08=4)=42.47%+34.11%=76.58% =P(Q5.02=4|Q4.08=5)+P(Q5.02=5|Q4.08=5)=24.21%+54.54%=78.75% Clear requirements reached between customer/users and development team: =P(Q5.02=4|Q5.03=4)+P(Q5.02=5|Q5.03=4)=56.79%+25.04%=81.83% =P(Q5.02=4|Q5.03=5)+P(Q5.02=5|Q5.03=5)=12.57%+61.12%=73.69%

Q4.08 Q5.03

Percent 78.77% 77.67% 77.67%

100

Percentage of Q5.02

90 80

80.1

77.5

76.6

78.8

81.8 73.7

70 60 50 40 30 20 10 Q4.02

Q4.08 Q5.02="4"

Q5.03

Q5.02="5"


So far, the critical path includes Q4.02ÆQ5.02ÆQ6.01. The next step was to trace which variable is the largest influencer of Q4.02. Candidate independent variables included Q3.02, “Sponsor throughout project”; Q3.03, “Sponsor was committed”; Q4.08, “Users made adequate time for requirements gathering” (see Figure 15). Table 41 summaries the results of this analysis of which variable had the greatest impact on Q4.02, as measured by the average percentage of projects that were considered to have high customer/user participation (Q4.02=4 or 5). These findings are also graphically depicted in Figures 19. No single variable exhibited an

137

overwhelming impact on high customer/user participation. However, given the data collected, Q4.08, “Users made adequate time for requirements gathering” had the largest impact on Q4.02 (74.90% average probability that Q4.02=4 or 5).

Table 41: Impact Summary For Q4.02, “Level of customer/user participation was high” Variable Q3.02 Q3.03 Q4.08

Average Percent

Question Sponsor throughout project: =P(Q4.02=4|Q3.02=4)+P(Q4.02=5|Q3.02=4)=39.50%+18.35%=57.85% =P(Q4.02=4|Q3.02=5)+P(Q4.02=5|Q3.02=5)=28.76%+47.28%=76.04% Sponsor was committed: =P(Q4.02=4|Q3.03=4)+P(Q4.02=5|Q3.03=4)=30.17%+41.54%=71.71% =P(Q4.02=4|Q3.03=5)+P(Q4.02=5|Q3.03=5)=32.99%+36.80%=69.79% Users made adequate time for requirements gathering: =P(Q4.02=4|Q4.08=4)+P(Q4.02=5|Q4.08=4)=47.11%+31.23%=78.34% =P(Q4.02=4|Q4.08=5)+P(Q4.02=5|Q4.08=5)=12.47%+58.98%=71.45%

66.95% 70.75% 74.90%

100

Percentage of Q4.02

90 76.0

80 70 60

78.3 71.7

69.8

71.5

57.9

50 40 30 20 10 Q3.02

Q3.03 Q4.02="4"

Q4.08

Q4.02="5"


The updated ‘critical path’ is Q4.08ÆQ4.02ÆQ5.02ÆQ6.01. The final piece is the lone independent variable of Q4.08, which is only influenced by one variable, namely Q3.02,

138

“Sponsor throughout project” (see Figure 18). The overall probability of Q4.08 being considered to be 4 or 5 (agree) is 70.12% (average of 75.53% and 64.70%). Table 42 summaries the results of this analysis of Q3.02, as measured by the average percentage of projects that were considered to have users that made adequate time available for requirements gathering (Q4.08=4 or 5).

Table 42: Impact Summary For Q4.08, “Customer/users made adequate time” Variable Q3.02

Question Sponsor throughout project: =P(Q4.08=4|Q3.02=4)+P(Q4.08=5|Q3.02=4)=38.20%+26.50%=64.70% =P(Q4.08=4|Q3.02=5)+P(Q4.08=5|Q3.02=5)=43.86%+31.67%=75.53%

Average Percent 70.12%

The completed path is Q3.02ÆQ4.08ÆQ4.02ÆQ5.02ÆQ6.01 (depicted with dashed lines in Figure 20). When the four variables along the critical path are forced to a response of 4, the probability of a project being considered a success is 79.42% (Q6.01=4 is 71.25% and Q6.01=5 is 8.17%). When these four variables are set to a response of 5, the probability increases to 95.27% (Q6.01=4 is 10.85% and Q6.01=5 is 84.42%). Based on the results of literature review and interviews with software practitioners previously presented, none of the items on the critical path were a surprise, including the two variables related to requirements (Q4.08 and Q5.02). The complete critical path, from start to finish, is as follows: •

Q3.02, “Sponsor throughout the project”.

•

Q4.08, “Users made adequate time for requirements gathering”.

•

Q4.02, “Level of customer/user participation was high”.

•

Q5.02, “Agreement on requirements was reached between customer/users”.

•

Q6.01, “Practitioners’ overall perception of project success”.

The entire critical path is composed of variables that were significant predictors of their respective dependent variables based on ordinal regression analysis (as shown in Section 8.2.3: Ordinal Regression Analysis). Figure 20 includes the critical path (dashed lines) and non-

139

significant regression relationships (bold lines). Most of the variables included in the model were significant predictors of their respective dependent variables, with the exception of the following independent/dependent relationships: •

Q5.01, “Functional scope changed” to Q5.05, “Well-defined software deliverables”.

•

Q4.02, “Level of customer/user participation was high” to Q5.03, “Requirements were clear and complete”.

•

Q3.03, “Sponsor was committed” to Q4.01, “High level of confidence in PM/team”.

•

Q4.03, “Participating customer/users stayed throughout” to Q6.01, “Practitioners’ overall perception of project success”.

•

Q4.01, “High level of confidence in PM/team” to Q6.01, “Practitioners’ overall perception of project success”.

Figure 20: Proposed Bayesian Belief Network With Critical Path (dashed lines) and Non-Significant Regression Paths (bold lines) A final investigation of the identified critical path was an ordinal regression analysis of each of the independent/dependent relationships along the path. Table 43 includes the associated Rsquare values, as well as the bivariate correlation (Spearman’s) coefficients and significance

140

levels (Section 8.2.2: Bivariate Correlation Analysis). The correlation coefficients along the critical path were classified as ‘low’, ranging from 0.12 to 0.37. See Appendix W for additional details.

Table 43: Summary of Ordinal Regression Models On Critical Path Dependent Variable Q4.08 Q4.02 Q5.02 Q6.01

Question User made adequate time for reqts. High customer/user participation. Agreement reached on requirements. Practitioner’s perception of success.

Cox and Snell R-square 0.15 0.37 0.20 0.12

Relationship and Correlation Coefficient (Significance Level) Q3.02ÆQ4.08: 0.31 (0.00) Q4.08ÆQ4.02: 0.58 (0.00) Q4.02ÆQ5.02: 0.43 (0.00) Q5.02ÆQ6.01: 0.53 (0.00)

As described earlier, the critical path identified above began with the final dependent variable, Q6.01, “Practitioners’ overall perception of project success” and its independent variables (those with direct relationships), and then working ‘upstream’ from that variable. A ‘what-if’ analysis was also conducted to determine which of the variables included in the model had the greatest impact, either direct or indirect, on Q6.01, “Practitioners’ overall perception of project success”. Similar to the above analysis, each variable was forced to a value of 4 and 5 to examine the effect on Q6.01. The average of these two probabilities was then calculated. The difference between the analysis and the development of the critical path was that all variables were considered, one at a time, without regard to the direct (independent/dependent) relationships among the variables. No single variable exhibited an overwhelming impact on practitioners’ perception of project success. However, given the data collected, Q5.02, “Agreement on requirements reached between customer/users and the development team” had the largest impact on Q6.01 (74.13% average probability that Q5.02=4 or 5), which also happened to have a direct causal connection to Q6.01 (see Figure 15). See Table 44 for a summary of the results and Appendix W for complete details.

141

Table 44: Summary of Variables’ Impact On Project Success Variable Q3.02 Q3.03 Q4.01 Q4.02 Q4.03 Q4.08 Q5.01 Q5.02 Q5.03 Q5.05

Question Sponsor throughout project: =P(Q6.01=4|Q3.02=4)+P(Q6.01=5|Q3.02=4)=41.18%+17.22%=58.40% =P(Q6.01=4|Q3.02=5)+P(Q6.01=5|Q3.02=5)=27.34%+46.05%=73.39% Sponsor was committed: =P(Q6.01=4|Q3.03=4)+P(Q6.01=5|Q3.03=4)=30.57%+35.65%=66.22% =P(Q6.01=4|Q3.03=5)+P(Q6.01=5|Q3.03=5)=30.18%+35.98%=66.16% Customer/users had high level of confidence in PM/development team: =P(Q6.01=4|Q4.01=4)+P(Q6.01=5|Q4.01=4)=35.54%+33.44%=68.98% =P(Q6.01=4|Q4.01=5)+P(Q6.01=5|Q4.01=5)=27.13%+42.76%=69.89% High customer/users participation: =P(Q6.01=4|Q4.02=4)+P(Q6.01=5|Q4.02=4)=39.50%+31.08%=70.58% =P(Q6.01=4|Q4.02=5)+P(Q6.01=5|Q4.02=5)=23.71%+49.27%=72.98% Participating customer/users stayed throughout project: =P(Q6.01=4|Q4.03=4)+P(Q6.01=5|Q4.03=4)=37.28%+21.80%=59.08% =P(Q6.01=4|Q4.03=5)+P(Q6.01=5|Q4.03=5)=27.53%+45.61%=73.14% Users made adequate time for requirements gathering: =P(Q6.01=4|Q4.08=4)+P(Q6.01=5|Q4.08=4)=33.84%+36.90%=70.74% =P(Q6.01=4|Q4.08=5)+P(Q6.01=5|Q4.08=5)=24.61%+44.94%=69.55% Functional scope change: =P(Q6.01=4|Q5.01=4)+P(Q6.01=5|Q5.01=4)=30.02%+33.78%=63.80% =P(Q6.01=4|Q5.01=5)+P(Q6.01=5|Q5.01=5)=29.67%+33.76%=63.43% Agreement on requirements reached between customer/users: =P(Q6.01=4|Q5.02=4)+P(Q6.01=5|Q5.02=4)=46.18%+24.96%=71.14% =P(Q6.01=4|Q5.02=5)+P(Q6.01=5|Q5.02=5)=17.88%+59.24%=77.12% Requirements were clear and complete: =P(Q6.01=4|Q5.03=4)+P(Q6.01=5|Q5.03=4)=36.63%+32.78%=69.41% =P(Q6.01=4|Q5.03=5)+P(Q6.01=5|Q5.03=5)=20.26%+49.75%=70.01% Requirements gathering resulted in well-defined software deliverables: =P(Q6.01=4|Q5.05=4)+P(Q6.01=5|Q5.05=4)=36.32%+34.74%=71.06% =P(Q6.01=4|Q5.05=5)+P(Q6.01=5|Q5.05=5)=20.70%+46.10%=66.80%

Average Percent 65.90% 66.19% 69.44% 71.78% 66.11% 70.15% 63.62% 74.13% 69.71% 68.93%

It is interesting to note the overall impact of various combinations of variables (i.e., those related to management/sponsor support and participation [Q3.02 and Q3.03], customer/user support and participation [Q4.01, Q4.02, Q4.03 and Q4.08], and requirements management [Q5.01, Q5.02, Q5.03 and Q5.05]) on successful projects (i.e., when Q6.01=4 or 5). For example, when all cases within the model were forced to a value of 4 for variables that are related to management/sponsor support and participation (Q3.02 and Q3.03), the probability of the project being considered a success was 61.81% (Q6.01 equaling 4=43.22% + Q6.01 equaling 5=18.59). Also, when all cases were forced to a value of 5 for Q3.02 and Q3.03, the probability of the project being considered a success was 77.54% (Q6.01 equaling 4=27.14% + Q6.01 equaling 5=50.40%). Similar, when all cases within the model are forced to a value of 4 for included variables that are related to

142

customer/user support and participation (Q4.01, Q4.02, Q4.03 and Q4.08), the probability of the project being considered a success was 77.49%. When all cases were forced to a value of 5 for Q4.02, Q4.03 and Q4.08, the probability of the project being considered a success was 90.28%. When all cases within the model are forced to a value of 4 for included variables that are related to requirements management (Q5.01, Q5.02, Q5.03 and Q5.05), the probability of the project being considered a success was 85.87%. When all cases were forced to a value of 5 for Q5.01, Q5.02, Q5.03 and Q5.05, the probability of the project being considered a success was 89.60%. See Table 45 and Figure 21 for details.

Table 45: Summary of Project Components’ Impact On Project Success Survey Section Management Support and Participation: =P(Q6.01=4|Q3.02=4, Q3.03=4)+P(Q6.01=5|Q3.02=4, Q3.03=4)=43.22%+18.59%=61.81% =P(Q6.01=4|Q3.02=5, Q3.03=5)+P(Q6.01=5|Q3.02=5, Q3.03=5)=27.14%+50.40%=77.54% Customer Support and Participation: =P(Q6.01=4|Q4.01=4, Q4.02=4, Q4.03=4, Q4.08=4)+ P(Q6.01=5|Q4.01=4, Q4.02=4, Q4.03=4, Q4.08=4)=62.82%+14.67%=77.49% =P(Q6.01=4|Q4.01=5, Q4.02=5, Q4.03=5, Q4.08=5)+ P(Q6.01=5|Q4.01=5, Q4.02=5, Q4.03=5, Q4.08=5)=18.92%+71.36%=90.28% Requirements Management: =P(Q6.01=4|Q5.01=4, Q5.02=4, Q5.03=4, Q5.05=4)+ P(Q6.01=5|Q5.01=4, Q5.02=4, Q5.03=4, Q5.05=4)=55.78%+30.09%=85.87% =P(Q6.01=4|Q5.01=5, Q5.02=5, Q5.03=5, Q5.05=5)+ P(Q6.01=5|Q5.01=5, Q5.02=5, Q5.03=5, Q5.05=5)=15.04%+74.56%=89.60%

Average Percent 69.68%

83.89%

87.74%

143

100

Percentage of Q6.01

90 80 70

87.7

83.9 69.7

60 50 40 30 20 10 Management Support & Participation:

Customer Support & Participation:

Requirements Management:

Figure 21: Percentage of Q6.01=4 or 5 When All Cases of Each Project Component=4 or 5.

This section of analysis and findings concludes with a brief summary of the major findings from each analytical method. Section 8.2.1 presented details of evidence that the survey instrument had a reasonable degree of internal consistency regarding the concepts it assumed to examine. There was also some evidence of identifiable multi-dimensionality (‘sub-constructs’) within each section of the survey. Section 8.2.2 presented evidence of significant bivariate correlations for each of the proposed independent-dependent variable relationships within the causal model, which serves to provide support for appropriateness of the model. Section 8.2.3 presented the results of the ordinal regression analysis, which provided a relatively large range of explanatory value among all of the eight causal relationships that combined to form the overall model. In addition, many independent variables proved to not be significant predictors of their associated dependent variable. Section 8.2.4 presented the causal model as a Bayesian Belief Network populated with observed frequencies that were based on the eight casual relationships. The variable with the largest direct impact on practitioners’ overall perception of project success was

144

agreement on requirements between customer/users and the development team. Figure 21 shows the variables associated with requirements management had the greatest impact, as a group, on project success.

The next chapter presents the conclusions to this study and recommendations for further study.

145

9. Conclusions

This study addresses some of the long-standing problems related to the software development process. Even if projects are completed, the software industry has a history of delivering late, expensive systems that often do not meet customer/users requirements (Chapter 1). Further, project managers are noted for under-managing some of the early non-technical components of the development process. These components tend to be more difficult to manage than technical components, and, consequently, are often largely unmanaged (Section 2.3). As a result, nontechnical issues more often plague software development than do technical problems. The categories of non-technical components investigated in this study are (1) sponsor/management support and participation, (2) customer/user support and participation, and (3) requirements management. The investigation of these non-technical components includes the following: •

An extensive literature review identified some early aspects of the software development process and some of their inter-relationships (Chapter 3).

•

Interviews with software practitioners (programmers, database developers, system analysts, etc.) identified aspects of the development process that practitioners considered important to project success.

The variables used in this investigation were found to be appropriate and useful in examining some of the important non-technical components of the software development process. Both the literature review and interviews with practitioners provided support for the creation of a survey and an associated causal model that included some of the early non-technical components of the software development process. Through the survey, software practitioners were asked to consider a particular project that they had recently worked on, with emphasis on several aspects related to sponsor/management, customer/users and requirements management. Respondents were asked to consider and rate the relative ‘success’ of that project. The research model developed is a Bayesian Belief Network, which is a probability-based model that graphically depicts causal

146

relationships between variables was developed. The model is based on a theorem developed in the 18th century by Thomas Bayes, which calculated probabilities that result from the values taken on by a causal chain of independent and dependent variables. (Today, modern computers, software applications and graphic user interfaces help make data analysis using Bayesian models quite straightforward. The Hugin Researcher 6.1 system was used to develop the Bayesian model for this study.) Graphically, oval nodes represent variables (independent and dependent), and arrows represent proposed causal relationships between these nodes (Section 8.1.4). A conditional probability table, which includes the probability of a particular variable equating to every possible value, underlies each node. Data analysis of the survey responses was used to validate the model that was developed.

In general, results from this study agree with the current body of published, largely anecdotal research. One of the major findings of this study is support for the critical nature of effective requirements gathering. While such support may not come as a surprise to many researchers and project managers, the quantitative nature of this study represents a first step in the validation of prior qualitative research. Further, a Bayesian Network proved to be a useful analytic tool for graphically depicting the probabilities associated with the investigated components of the software development process. The model was particularly useful in isolating the impact (both direct and indirect) of its component(s) on the likelihood of practitioners considering a project to be successful. The analysis identified a series of components (a ‘critical path’) that had the greatest impact on project success. In general chronological order, this path was made up of the following factors: 1. 2. 3. 4.

Having a sponsor throughout the project, Users who make adequate time for requirements gathering, A high level of customer/user participation in the development process, and Agreement on requirements between customer/users and the development team.

147

As noted earlier, the project variables included in this study were related to sponsor/management, customer/user participation and requirements management. When considered as a group, the requirements management variables exhibited the greatest impact on software project success when compared with those variables included within the other two groups. This was determined through isolating the independent impact of each of the three groups of variables on the probability of practitioners considering a project to be a success (Section 8.2.4). The following is a list of the project variables in the Bayesian model that are related to requirements management: • • • •

Change in functional scope. Agreement on requirements was reached between customer/users and the development team. Requirements were clear and complete (scope of project’s functionality was welldefined). Requirements gathering resulted in well-defined software deliverables (forms, reports, utilities, etc).

The impact on project success that was observed represents a combined result of the particular group of variables (nodes) that had both a direct and indirect causal relationship on project success according to the model. (Nodes with a direct connection to the project success node are shown through a single connecting arrow between the two nodes. Indirect connections were shown through a series of nodes and interconnecting arrows.) An example of an indirect influence on project success is shown in Figure 15 in Section 8.2.4: 1. Facilitating clear and complete requirements is shown to directly impact well-defined project deliverables. 2. Well-defined project deliverables contribute to a higher level of confidence by customer/users in the development team. 3. A high level of confidence then affects practitioners’ overall perception of project success. Agreement on requirements with customer/users, a high level of customer/user participation, and users that make adequate time for requirements gathering represented the individual components with the largest impact (either direct or indirect) on practitioners’ overall perception of project success (Section 8.2.4).

148

The model developed here, and its critical path, can contribute to a project manager’s awareness of the potential ‘downstream’ implications of their actions (or inactions) on the development process. As a result, the management (or lack of management) of components related to sponsor/champion, customer/users and requirements management can have major implications later in the development process. For example, securing a sponsor or champion early in the project has implications throughout the remainder of the project, including contributing to effective requirements gathering and high level of confidence by the customer/users in the development team.

The results of this study can also contribute to project managers’ understanding of the importance of software practitioners’ perception of project success. Practitioners do not necessarily place the same value on aspects of a given development project as does the organization or its management (Sections 2.1 and 4.2). In particular, management tends to place emphasis on delivering a product on time and within budget. (A concern for meeting customer/users’ requirements is also common.) However, practitioners have been shown to not place as much emphasis on these factors. Rather, practitioners tend to value aspects of the development process that are related to the following: • •

Their professional/personal experience and growth, including a sense of achievement and of doing a good job, Project planning and requirements, including having a well-planned project and requirements that are accepted by the development team as being realistic and achievable) (Section 4.2).

By basing the concept of project success on what practitioners report is important to their perception of success, this research has identified those aspects of the development process that make working on a project more pleasurable and motivating for developers. Further, motivation is the single greatest contributor to software development productivity. Productivity, in turn, has

149

direct implications for the organization’s ability to effectively and efficiently develop software, which includes delivering completed systems that meet customer/user requirements on-time and within budget. (Section 2.1).

Project managers can use the findings of this study to increase the probability that the projects they manage will be successful, both from an organizational and practitioner perspective. As noted earlier, management of requirements is critical successful software development. More specifically, this study found the following.

(1) Having a project sponsor/companion, or a 'champion of the cause', will, in turn, support the involvement of customers/users during the development process. A sponsor is particularly valuable when he/she encourages the customer/users to spend adequate time developing requirements. If a project sponsor is not available at the beginning of a project, the project manager should try to get one as soon as possible. Since a sponsor is generally from uppermanagement, he/she can act as an effective buffer between the project team and unrealistic pressure to deliver increased functionality and/or a project faster than originally planned. Such pressures may originate from the customer/users, and then come to the attention of uppermanagement. If a sponsor cannot be found at all, then the project has a higher likelihood of failure. A project manager must then work particularly hard to establish good communication between the development team and customer/users (see #2 and #3 below). This will help to foster understanding of the time necessary to deliver the agreed functionality. As a result, it may help to alleviate subsequent customer/user pressure and/or unrealistic demands.

(2) A project manager who encourages high customer/user participation in the development process is more likely to achieve a higher level of agreement on requirements between

150

customer/users and the developers. Managers can expect this agreement to contribute to clearer and more complete requirements, and ultimately to better-defined software deliverables. A project manager who communicates with both the development team and customer/users helps to facilitate customer/user participation. This will help managers to understand when (or perhaps more importantly, if) these two groups of stakeholders perceive that an 'adequate' amount of time has been spent on requirements gathering. If the development team and customer/users spend adequate time developing requirements, project managers can expect a greater likelihood that these two groups will ultimately reach agreement on the functionality of the proposed system. However, it is important that a project manager communicates with both groups because each may have a quite different perception of what constitutes ‘adequate time’. A definition of adequate time is important because there may be significant differences in perceptions of ‘adequate time’ between these two groups. Managers may be able to gauge the meaning of 'adequate' time, through informal discussion, surveys and/or semi-structured interviews. If it is determined that adequate time has not be spent on requirements gathering, project managers may need to meet with their development team and customer/users independently, and perhaps together afterwards, in order to determine where more time and attention needs to be spent by either, or both, parties.

(3) Early and on-going communication among the project manager, development team and customer/users is closely related to customer/users reaching an agreement with the development team regarding system functionality. Managers can expect this agreement to result in fewer misunderstandings of the functionality of the final product, and consequently less rework in the later stages of the development process.

151

This section concludes with the meta-level contributions of this study to the domain of the software engineering. •

A working definition of project success from the software practitioner perspective (it is anticipated that this perspective will be used in further study) (Section 4.2).

•

A survey instrument useful in the evaluation of early components of the software development process. This instrument can serve as a model for the investigation additional components of the development process (estimation, scheduling, testing, etc.) (Section 6.3).

•

A probability-based model (Bayesian Belief Network) that graphically depicts some of the early aspects of the software development process. This model can also be used to conduct ‘what-if’ analysis in order to isolate important variables. Further, this model can be readily expanded to include additional components of the development process (see Chapter 10).

152

10. Recommendations For Further Study

Recommendations for further study are arranged as follows. First, some issues related to revising the survey instrument are presented, followed by ideas to expand on the proposed Bayesian Belief Network. Lastly, issues related additional data collection is discussed.

Two issues related to the survey instrument arose that would help to refine and focus the specific concept being investigated, and they are as follows: •

A survey respondent noted that his development team felt a sense of achievement after the completed project had been implemented. However, this study asked about a sense of achievement while working on a project in order to offer some insight into personal satisfaction of practitioners as a result of their professional involvement in the development of a project. This study did not consider any post-project sense of achievement, which should be considered in future work.

•

In order to lend some context to the idea of customer/users having a high level of confidence in the project manager/development team, an additional question should be considered in future work that addresses whether the organization (and/or development team) has done software development work for a particular customer in the past.

Since this was a preliminary effort at constructing and testing a Bayesian Belief Network related to the early aspects of the software development process, several issues arose regarding revising and/or expanding on the proposed model. They included the following: •

The use of Multicriteria Decision Aids (MCDA) should be considered in conjunction with BBN in order to include external criteria in the model (i.e. political, financial, environmental and technical). It is precisely these criteria that BBNs cannot readily handle.

•

Additional causal linkages among the variables already included within the proposed model should be investigated, as there was evidence of additional linkages among variables that were not included in the proposed model.

•

Additional variables could be added to the model, as appropriate, that were included in the survey instrument, but were not included in the proposed model. This could serve to create a more robust model.

•

Based on additional investigation (literature review and interviews with practitioners), new variables could be added to the model that were not included in survey. This, too, could serve to create a more robust model.

153

•

Factor analysis conducted with the five variables that were used to defined project success in this study provided evidence of two constructs, which were consistent with results from a pilot study. It would be interesting to investigate the relationship between these two ‘component definitions’ and variables related to management/sponsor, customer/users and requirements management. The identified constructs included “planning and requirements acceptance” and “personnel attributes”. This investigation could provide an analysis of project success on a more granular level than simply offering one definition of project success, as was the case with the proposed model.

•

The relationship between the proposed model and the organizational/managerial perspective of project success (completing a project that meets customer/user requirements and is delivered on time and within budget) should be investigated in order to simultaneously investigate organizational and practitioners’ perception of project success.

A few issues arose related to additional data collection in order to substantiate the generalizability of this study. •

This study was cross-sectional in nature (essentially a pre-experiment, case study design). Specifically, it is a case study, which is designed to provide a snapshot of the current state of the practice of software development. As such, these results will essentially provide findings that will be generalizable to the domestic software development community. Through selection of equivalent randomly selected sampled groups for use in further studies, we could further test the proposed correlation and final model for reliability and generalizability. This would also assist in evaluating changes and/or trends over time in the proposed prediction model.

•

Additional cases should be gathered that reflect relatively unscuccessful projects, as defined in this study, in order to provide a more even distribution of opinions.

•

Continued investigation should examine the variable(s) with the largest impact on software practitioners’ overall perception of project success.

154

List of References

Agarwal, Ritu and Thomas W. Ferratt, Recruiting, Retaining and Developing IT Professionals: An Empirically Derived Taxonomy of Human Resource Practices, CPR Conference, Boston, MA (1998) p. 292-302. Agena Ltd, Technical Report: Basics of Bayesian Probability, http://www.agena.co.uk (February 2, 2002a) Agena Ltd, Technical Report: Basics of Bayesian Networks, http://www.agena.co.uk (February 11, 2002b) Amoako-Gyampah, Kwasi and White, Kathy B., When Is User Involvement Not User Involvement, Information Strategy: The Executive’s Journal, Volume 13, Number 4 (Summer 1997) p. 40-45. Anderson, H. and Narshimhan, R. Assessing Project Implementation Risk: A Methodological Approach, Management Science, Volume 25 (1979) p. 512-521. Babbie, Earl, The Practice of Social Research (9th Edition) Wadsworth/Thomson Learning, Belmont, California (2001). Baccarini, David, The Logical Framework Method For Defining Project Success, Project Management Journal, Volume 30, Number 4 (December 1999) p. 25-32. Bagozzi, Richard P. and Youjae Yi, On The Evaluation of Structural Equation Models, Journal of The Academy of Marketing Science, Volume 16, Number 1 (Spring 1988) p. 74-94. Bailey, J.E. and S.W. Pearson, Development of A Tool For Measuring and Analyzing Computer User Satisfaction, Management Science, Volume 29, Number 5 (1983) p. 530-545. Barki, Henri, Suzanne Rivard and Jean Talbot, Toward An Assessment of Software Development Risk, Journal of Management Information Systems, Volume 10, Number 2 (Fall 1993) p. 203223. Baroudi, Jack J. and Wanda J. Orlikowski, A Short-Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes On Use, Journal of Management Information Systems, Volume 4 (Spring 1988) p. 44-59. Boehm, Barry, Software Engineering Economics, Prentice Hall, Englewood Cliffs, NJ (1981). Boehm, Barry W., Software Risk Management, IEEE Computer Society Press, Los Altamitos, CA (1989). Boehm, Barry, Software Risk Management: Principles and Practices, IEEE Software, Volume 8, Number 1 (January 1991) p. 32-41.

155

Boehm, Barry, Project Termination Doesn’t Equal Project Failure, IEEE Computer, Volume 33, Number 9 (September 2000) p. 94-96. Boehm, Barry and Victor R. Basili, Software Defect Reduction Top 10 List, IEEE Computer, Volume 34, Number 1 (January 2001) p. 135-137 Boehm, Barry and Alexander Egyed, Software Requirements Negotiation: Some Lessons Learned, International Conference on Software Engineering (1998) p. 503-506. Boehm, Barry and Tom DeMarco, Software Risk Management, IEEE Software, Volume 14, Number 3 (May/June 1997) p. 17-19. Brady, Sheila and Tom DeMarco, Management-Aided Software Engineering, IEEE Software, Volume 11, Number 6 (November 1994) p. 25-32. Brooks, Frederick P., Jr., The Mythical Man-Month, Addison-Wesley, Reading, MA (1995). Bytheway, Andrew J., Successful Software Projects and How To Achieve Them, IEEE Software, Volume 16, Number 3 (May/June 1999) p. 15-17. Cook, T.D. and D.T. Campbell, Quasi Experimentation: Design and Analytical Issues For Field Settings, Rand McNally, Chicago, IL (1979). Chatzoglou, Prodromos D., Factors Affecting Completion of the Requirements Capture Stage of Project With Different Characteristics, Information and Software Technology, Volume 39, Number 9 (September 1997) p. 627-640. Choe, Jong-Min, The Effects of User Participation on the Design of Accounting Information Systems, Information & Management, Volume 34, Number 3 (1998) p. 185-198. Clavadetscher, Carl, User Involvement: Key To Success, IEEE Software, Volume 15, Number 2 (March/April 1998) p. 30, 32. Couger, Daniel, Motivators vs. Demotivators In The IS Environment, Journal of Systems Management, Volume 39, Number 6 (June 1988) p. 36-41. Curtis, Bill, Measurement and Experimentation In Software Engineering, Proceedings of The IEEE, Volume 68, Number 9 (September 1980) p. 1144-1157. Curtis, Bill, William Hefley and Sally Miller, Overview of The People Capability Maturity Model (P-CMM), www.sei.cmu.edu/publications/documents/95.reports/95.mm.001.html, accessed December 18, 2001. Davis, G.B., Strategies For Information Requirements Determination, IBM System Journal, Volume 21 (1982) p. 4-30. Davis, F. D., Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology, MIS Quarterly, Volume 13, Number 3(September 1989) p. 319-339.

156

DeLone, W.H. and E.R. McLean, Information Systems Success: The Quest For The Dependent Variable, Information Systems Research, Volume 3 (1992) p. 60-95. DeMarco, Tom, Non-Technical Numbers In Software Engineering, IEEE Conference on Software Engineering (1991) p.149-150. DeMarco, Tom and Timothy Lister, Peopleware: Productive Projects and Teams (2nd Edition), Dorset House Publishing Co., New York, NY (1999). Ein-Dor, P. and E. Segev, Organizational Context & MIS Structure: Some Empirical Evidence, MIS Quarterly, Volume 6 (1982) p. 55-68. Ellis, T. Selwyn and Robert L. Webster, IS Managers’ Innovation Toward Telecommuting: A Structural Equation Model, Preceedings of The 31st Hawaii International Conference On System Sciences, Volume 4 (January 6-9, 1998) p. 161-168. Ewusi-Mensah, Kweku, Critical Numbers In Abandoned Information Systems Development Projects, Communications of The ACM, Volume 40, Number 9 (September 1997) p. 74-80. Evanco, William, J. Drew Procaccino and June M. Verner, submitted to Journal of Software Engineering (Summer 2002). Fenton, Norman E. and Martin Neil, Software Metrics: A Roadmap, Future of Software Engineering (FoSE) at ICSE'00, Limerick, Ireland (June 4-11, 2000a). Fenton, Norman and Martin Neil. Making Decisions: Using Bayesian Nets and MCDA (2000b). Fienberg, S.E. and W. Mason, Identification and Estimation of Age, Period and Cohort Models In The Analysis of Discrete Archival Data, Sociological Methodology (1979) p. 1-67. Ferratt, Thomas W., Ritu Agarwal, Jo Ellen Moore and Carol V. Brown, Observations From “The Front”: IT Executives on Practices To Recruit and Retain Information Technology Professionals, SIGCPR Conference, New Orleans, LA (1999) p.102-112. Gefen, David, Detmar W. Straub and Marie-Claude Boudreau, Structural Equation Modeling and Regression: Guidelines For Research Practice, Communications of The Association For Information Systems, Volume 4, Number 7 (August 2000). Gefen, David and Detmar W. Straub, The Relative Importance of Perceived Ease-of-Use in IS Adoption: A Study of E-Commerce Adoption, JAIS, Volume 1, Number 8 (2000) p. 1-30. Glass, Robert L., Building Quality Software, Prentice-Hall, Englewood Cliffs, NJ (1992) p. 253. Glass, Robert L., The Software Research Crisis, IEEE Software, Volume 11, Number 6 (November 1994) p. 42-47. Glass, Robert L., Software Runaways, Prentice-Hall, Upper Saddle River, NJ (1998). Glass, Robert L., Evolving A New Theory of Project Success, Communications of The ACM, Volume 42, Number 11 (November 1999) p. 17.

157

Glass, Robert L., Frequently Forgotten Fundamental Facts About Software Engineering, IEEE Software, Volume 18, Number 3 (May/June 2001a) p. 110-112. Glass, Robert L., ComputingFailure.com: War Stories From The Electronic Revolution, Prentice Hall PTR, Upper Saddle River, NJ (2001). Ginzberg, M.J., Early Diagnosis of MIS Implementation Failure Promising Results and Unanswered Questions, Management Science, Volume 27, Number 4 (1981) p. 459-478. Guilford, J.P., Fundamental Statistics In Psychology & Education, McGraw-Hill, New York, NY (1956). Haberman, S.J., Analysis of Frequency Data, University of Chicago Press (Chicago, Illinois) (1974). Hair, Joseph F., Rolph E. Anderson, Ronald L. Tatham and William C. Black, Multivariate Data Analysis With Readings, 3rd Edition, Macmillan Publishing Company (New York, NY) 1992. Hannah, Murray and Paul Quigley, Presentation of Ordinal Regression Analysis On The Original Scale, Biometrics, Volume 52 (June 1996) p. 771-775. Herzberg, Frederick, One More Time: How Do You Motivate Employees?, Harvard Business Review, (September/October 1987) p. 109-120. Hohmann, L., Journey of The Software Professional: The Sociology of Software Development, Prentice-Hall, Upper Saddle River, NJ (1997) Hohmann, L., Coaching The Rookie Manager, IEEE Software, Volume 16, Number 1 (January/February 1999) p. 16-19. Humphreys, Watts S., Managing the Software Process, Addison-Wesley Publishing Company, Reading MA (1990). Hunton, J.E. and J.D. Beeler, Effects of User Participation in Systems Development: A Longitudinal Field Experiment, MIS Quarterly, Volume 21, Number 4 (1997) p. 359-388. Igbaria, M. and J.J. Baroudi, The Impact of Job Performance Evaluations on Career Advancement Prospects: An Examination of Gender Differences In The IS Workplace, MIS Quarterly, Volume 19, Number 1 (1995) p. 107-123. Interviews (semi-structured) with fifteen (15) software practitioners, conducted at Drexel University (Philadelphia, PA), Spring 2001. Ives, B., M.H. Olson and J. Baroudi, The Measurement of User Information Satisfaction, Communications of The ACM, Volume 26, Number 10 (1983) p. 785-793. Jiang, J., G. Klein and J. Balloun, Ranking of System Implementation Success Factors, Project Management Journal, Volume 27, Number 4 (December 1996) p. 50-55.

158

Jiang, James and Gary Klein, Software Development Risks To Project Effectiveness, Journal of Systems and Software, Volume 52, Number 1 (2000) p. 3-10. Jiang, James J., Gary Klein and Richard Discenza, Perception Differences of Software Success: Provider and User Views of System Metrics, The Journal of Systems and Software (accepted July 2, 2001). Jones, Casper, Patterns of Large Software Systems: Failure and Success, IEEE Computer, Volume 28, Number 3 (March 1995) p. 86-87. Kellner, Marc I., Non-Technical Issues In Software Engineering (Panel Session Overview), IEEE Conference on Software Engineering (1991) p.149-150. Keider, Stephen P., Why Projects Fail, Datamation (December 1974) p. 59-61. Keil, Mark, Linda Wallace, Dan Turk, Gayle Dixon-Randall and Urban Nulden, An Investigation of Risk Perception and Risk Propensity On The Decision To Continue A Software Development Project, The Journal of Systems and Software, Volume 53, Number 2 (August 31, 2000a) p. 145157. Kerzner, H., Project Management, Van Nostrand Reinhold (1989). Kettinger, W.J. and C.C. Lee, Perceived Service Quality and User Satisfaction With The Information Services Function, Decision Sciences, Volume 25, Number 5/6 (1994) p. 737-766. Kiess, Harold O., Statistical Concepts For The Behavior Sciences (2nd Edition), Allyn & Bacon (1996). Klein, Gary and James J. Jiang, Seeking Consonance In Information Systems, The Journal of Systems and Software, Volume 56 (2001) p. 195-202. Leffingwell, Dean and Don Widrig, Managing Software Requirements: A Unified Approach, Addison-Wesley (2000). Linberg, Kurt R., Software Developer Perceptions About Software Project Failure: A Case Study, The Journal of Systems and Software, Vol. 49, Issue 2/3 (December 30, 1999) p. 177-192. Lu, H-P. and Wand, J-Y., The Relationship between Management Styles, User Participation, and Systems Success over MIS Growth Stages, Information & Management, Volume 32, Number 4 (1997) p. 203-213. Lucas, H.C., Jr., The Use of Interactive Information Storage and Retrieval System In Medical Research, Communications of The ACM, Volume 21, Number 3 (1978) p. 197-205. Lyons, Michael, The DP Psyche, Datamation, Volume 31, Number 16 (August 15, 1985) p. 103105. Lyytinen, Kalle, Expectation Failure Concept and System Analysts’ View of Information System Failures: Results of an Exploratory Study, Information and Management, Volume 14 (1988) p. 45-56.

159

Maddala, G.S., Limited-Dependent and Qualitative Variables In Economics, Cambridge University Press, Cambridge, UK (1983). McConnell, Steve, Rapid Development, Microsoft Press, Redmond, Washington (1996). McCullagh, Peter, Regression Models For Ordinal Data, Journal of The Royal Statistics Society, Series B, Volume 42, Number 2 (1980) p. 109-142. McFarlan, F.W., Portfolio Approach To Information Systems, Harvard Business Review, Volume 59 (1981) p.142-150. McKeen, J.D. and T. Guimares, Successful Strategies for User Participation in Systems Development, Journal of Management Information Systems, Volume 14, Number 2 (Fall 1997) p. 133-150. McLean, E.R., S.J. Smits and J.R. Tanner, The Importance of Salary On Job and Career Attitudes of Information Systems Professionals, Information & Management, Volume 30, Number 6 (1996) p. 291-299. Moynihan, Tony, Coping With Requirements-Uncertainty, The Theories-of-Action of Experienced IS/Software Project Managers. Journal of Systems and Software, Volume 53, Number 2 (August 31, 2000) p. 99-109. Moynihan, Tony, Coping With Client-Based 'People-Problems': The Theories-of-Action of Experienced IS/Software Project Managers. Journal of Systems and Software, Volume 39 (2002) p. 377-390. Niedermayer, Daryle, An Introduction To Bayesian Networks and Their Contemporary Applications, http://www.gpfn.sk.ca/~daryle/papers/bayesian_networks.bayes.html (1998) visited June 2001. Nidumolu, Sarma R., Standardization, Requirements Uncertainty and Software Project Performance, Information & Management, Volume 31, Number 3 (1996) p. 135-150. Nunnally, J.C., Psychometric Theory, McGraw-Hill, New York, NY (1967). Nunnally, J.C., Psychometric Theory, 2nd Edition, McGraw-Hill, New York, NY (1978). Nunnally, J.C. and I.H. Bernstein, Psychometric Theory, 3rd Edition, McGraw-Hill, New York, NY (1994). Peter, L.J. and R. Hull, The Peter Principle, Williams Morrow, New York, NY (1969). Pinto, Jeffrey K. and Samuel J. Mantel, Jr., The Causes of Project Failure, IEEE Transactions on Engineering Management, Volume 37, Number 4 (November 1990) p. 269-276. Pinto, Jeffrey K. and D.P. Slevin, Project Success: Definitions and Measurement Techniques, Project Management Journal, Volume 19, Number 1 (1988) p. 67-72.

160

Powers, R.F. and G.W. Dickson, MIS Project Management: Myths, Opinions & Reality, California Management Review, Volume 15, Number 3 (1973) p. 147-156. Pressman, Roger, Software Process Impediment, IEEE Software, Volume 13, Number 5 (September 1996) p. 16-17. Pressman, Roger, Software Engineering: A Practitioner’s Approach, McGraw-Hill, New York, NY (1997). Pressman, Roger, Fear of Trying: The Plight of Rookie Project Managers, IEEE Software, Volume 15, Number 1 (January/February 1998) p. 50-51, 54. Procaccino, J. Drew and June M. Verner, Early Risk Factors For Software Development, European Conference on Cost Estimation, London, England (April 2001a). Procaccino, J. Drew and June M. Verner, Pilot Study #1: Software Practitioner’s Success Factors (April 2001b). Procaccino, Drew and June Verner, Software Practitioner’s Perception of Project Success: A Pilot Study, International Journal of Computers, The Internet & Management, Volume 10, Number 1 (January-April 2002) p. 20-30. Procaccino, Drew, June Verner, Scott Overmyer and Marvin Darter, Case Study: Factors For Early Prediction of Software Development Success, Journal of Information & Software Technology, Volume 44 (January 15, 2002a) p. 53-62. Procaccino, Drew, June Verner and William Evanco, Software Development Risk Analysis: An Exploratory Study, work-in-progress (Winter 2002b). Rai, A., H. Song and M. Troutt, Software Quality Assurance: An Analytical Survey and Research Prioritization, The Journal of Systems and Software, Volume 40, Number 1 (1998) p. 67-84. Rainer, R. Kelly and Hugh J. Watson, The Keys To Executive Information Systems Success, Journal of Management Information Systems, Volume 12, Number 2 (Fall 1995) p. 83-98. Rasch, Ronald H., & Henry L. Tosi, Factors Affecting Software Developers' Performance: An Integrated Approach, MIS Quarterly, Volume 16, Number 3 (September 1992) p. 395-413. Ravichandran, T. and Arun Rai, Quality Management In Systems Development: An Organizational System Perspective, MIS Quarterly. Volume 24, Number 3 (September 2000) p. 381-416. Ridings, Catherine M. and Lauren B. Eder, The Determinants of Job Satisfaction For IS Professionals In Technical Career Paths, CPR Conference, Boston, MA (1998) p. 170-173. Rittel, Horst, Second-Generation Design Methods, N. Cross (Ed.), Developments in Design Methodology, John Wiley and Sons, New York, NY (1984) p. 317-327.

161

Robey, Daniel, Larry A. Smith and Leo R. Vijayasarathy, Perceptions of Conflict and Success In Information Systems Development Projects, Journal of Management Information Systems, Volume 10, Number 1 (Summer 1993) p. 123+. Saleem, Naveed, An Empirical Test of the Contingency Approach to User Participation in Information Systems Development", Journal of Management Information Systems, Volume 13, Number 1 (Summer 1996) p. 145-166. Santos, J. Reynaldo A., Cronbach’s Alpha: A Tool For Assessing The Reliability of Scales, Journal of Extension, Volume 37, Number 2 (April 1999), http://www.joe.org/joe/1999april/tt3.html. Schenk, K.D and Vitalari, N.P., et al., Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do?, Journal of Management Information Systems, Volume 15, Number 1 (Summer 1998) p.9-51. Schumacker, Randall E. and Richard G. Lomax. A Beginner’s Guide To Structural Equation Modeling. Lawrence Erlbaum Associates. Mahwah, NJ (1996). Standish Group, CHAOS, http://www.standishgroup.com/visitor/voyages.htm, 1995a. Standish 1995b.

Group,

Unfinished Voyages, http://www.standishgroup.com/visitor/voyages.htm,

Sumner, Mary, Critical Success Factors In Enterprise Wide Information Management Systems Projects, SIGCPR Conference, New Orleans, LA (1999) p. 297-303. Tackett, Buford D. and Buddy Van Doren, Process Control For Error-Free Software: A Software Success Story, IEEE Software, Volume16, Number 3 (May/June 1999) p. 24-29. Verner, J. M., Overmyer, S. P. and McCain, K.W., In The 25 Years Since The Mythical ManMonth What Have We Learned About Project Management?, Information and Software Technology, Volume 41, Number 14 (November 5, 1999) p. 1021-1026. Verner, J. M. and Cerpa, work-in-process Summer 2002. Warne, Leoni and Dennis Hart, The Impact of Organizational Politics on Information Systems Project Failure – A Case, Proceedings of The 29th Annual Hawaii International Conference On System Sciences), IEEE (1996) p.191-201. Wateridge, John, How Can IS/IT Projects Be Measured For Success?, International Journal of Project Management, Volume 16, Number 1 (1998) p. 59-63. Watson, R.T., L.F. Pitt and C.B. Kavan, Measuring Information Systems Service Quality: Lessons From Two Longitudinal Case Studies, MIS Quarterly, Volume 22, Number 1 (March 1998) p. 61-79. Wohlin, C., A. von Mayrhauser, M. Host and B. Regnell, Subjective Evaluation As A Tool For Learning From Software Project Success, Information and Software Technology, Volume 42 (2000) p. 983-992.

162

Wynne, B., Measuring The Immeasurable or Credibility In The Public Sector, Interfaces, Volume 8, Number 1 (1977) p. 106-109. Zmud, Robert W., Management of Large Software Development Efforts, MIS Quarterly (September 1980) p. 45-55. E-mail received from Bob Glass, October 3, 2001 (04:59:11 pm). E-mail received from Warren Harrison, October 5, 2001 (12:51:19 am). E-mail received from Steve McConnell, October 5, 2001 (08:34:19 pm).

163

Appendix A: Various Perspectives on Defining Software Development Project Success Process-Related: • Development team’s overall productivity and level of effectiveness regarding its interactions with non-team members [Robey, et al 1993]. •

Practitioner’s perception of process [Glass, 1999; Linberg 1999; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Stakeholder participation, including top management (TQM-based) [Ravichandran and Rai 2000].

•

Stakeholder satisfaction with project management [Baccarini 1999].

•

User participation in process [Baroudi and Orlikowski 1988].

Product-Related: • Completed Project (general): o All stakeholders’ satisfaction with completed system [Anderson 1999; Bytheway 1999]. o

Amount of system usage by users [Ginzberg 1981, also cited Lucas 1978, Wynne 1977].

o

Client satisfaction [Pinto and Slevin 1988].

o

Customer satisfaction (TQM-based) [Ravichandran and Rai 2000].

o

Customer satisfaction (“customer satisfaction = meets requirements + sufficient quality + delivered when needed + affordable cost. Note that this equation decouples cost and schedule from estimates for them”) [e-mail: Glass 2001]

o

“Efficacy in meeting a project's goals, however those goals are defined”, which may include goals of maximizing functionality, minimizing program errors, minimizing cost, minimizing time to complete [e-mail: McConnell 2001].

o

Implementation of completed system [Pinto and Mantel 1990].

o

Meeting or exceeding user’s expectations [Linberg 1999].

o

On time delivery, delivered within budget and meets business objectives/requirements [Keider 1974; Lyytinen 1988; Pinto and Slevin 1988; Barki, et al 1993; Standish Group 1995a; Brooks 1995; Jones 1995; Nidumolu 1996; Amoako-Gyampah, et al 1997; Baccarini 1999; Linberg 1999; Jiang and Klein 2000; Leffingwell and Widrig 2000; Jiang, et al 2001].

o

On time delivery, delivered within budget, meets business objectives/requirements, achieves purpose, happy users, meets quality objectives [Wateridge 1998].

o

On time delivery, delivered within budget, meets business objectives/requirements, minimum/mutually agreed upon changes in scope, no distributing flow of organizational work, no change to corporate culture [Wateridge 1998 cited Kerzner 1989].

o

Practitioner’s perception of completed system [Glass 1999; Linberg 1999].

o

Practitioner’s perception of system usefulness [Pinto and Mantel 1990].

164

•

•

o

Practitioner’s (team) compliance with budget and schedule [Robey, et al 1993].

o

Product’s impact on customer service [Watson, et al 1998].

o

Product’s impact on effectiveness of business operations [Watson, et al 1998].

o

Service quality provided for system [Kettinger and Lee 1994; Watson, et al 1998].

o

Timeliness (related to market window) of system delivery [e-mail: Warren 2001]

o

Use of system (extent of use) [Davis 1989; Gefen and Straub 2000].

o

User satisfaction with system performance [Ginzberg 1981].

o

User satisfaction with completed system [Bailey and Pearson 1983 cite Powers and Dickson 1973; Pinto and Mantel 1990; Kettinger and Lee 1994; Baroudi and Orlikowski 1988; Bailey and Pearson 1983; Jiang, et al 2001].

Output: o Consumption of system output by recipient [cited from Ein-Dor and Segev 1982]. o

Quality of output, related to reliability of information produced [Wohlin, et al 2000; Jiang and Klein 2000; Rai, et al 1998; Tackett and VanDoren 1999; Ravichandran and Rai 2000; Jones 1995; Pressman 1997].

o

User perception of quality of information provided by system [Ives, et al 1983].

Cancelled Project: o Project cancellation and contributing issues [Ewusi-Mensah 1997]. o

Project cancellation and relationship to project failure [Boehm 2000].

Other: • Differences between practitioners and end-user perspectives [Jiang, et al 2001]. •

Differences among all project stakeholders’ perception of success due to various met or unmet expectations [Robey, et al 1993].

•

Practitioner’s perception of success (general) [Linberg 1999; Verner, et al 1999; Procaccino and Verner 2001a; Procaccino, et al 2002a; Procaccino, et al 2002b].

•

Numerous perspectives (system quality, information quality, use, user satisfaction, individual impact and organizational impact) [DeLone and McLean 1992; includes numerous success-related literature citations].

•

Numerous perspectives (time to develop project, cost of project, project performance, client use of system, client satisfaction with system and organizational effectiveness [Pinto and Slevin 1988].

165

Appendix B: Summary Results of The Standish Group’s CHAOS Study [1995a] Successful Projects Rank 1 2 3 4 5 6 7 8 9 10

Project Success Factors User Involvement Executive Management Support Clear Statement of Requirements Proper Planning Reali Expectations Smaller Project Milestones Competent Staff Ownership Clear Vision & Objectives Hard-Working, Focused Staff Other

Pct 15.9 13.9 13.0 9.6 8.2 7.7 7.2 5.3 2.9 2.4 13.9

Challenged Projects Rank 1 2 3 4 5 6 7 8 9 10

Project Challenged Factors Lack of User Input Incomplete Requirements & Specs Changing Requirements & Specs Lack of Executive Support Technology Incompetence Lack of Resources Unrealistic Expectations Unclear Objectives Unrealistic Time Frames New Technology Other

Pct 12.8 12.3 11.8 7.5 7.0 6.4 5.9 5.3 4.3 3.7 23.0

Impaired Projects Rank 1 2 3 4 5 6 7 8 9 10

Project Impaired Factors Incomplete Requirements Lack of User Involvement Lack of Resources Unrealistic Expectations Lack of Executive Support Changing Requirements & Specs Lack of Planning Didn't Need It Any Longer Lack of IT Management Technology Illiteracy Other

Pct 13.1 12.4 10.6 9.9 9.3 8.7 8.1 7.5 6.2 4.3 9.9

166

Appendix C: Results From Standish Group’s Unfinished Voyages [1995b] Project Success Potential User Involvement • Do I have the right user(s)? • Did I involve the user(s) early and often? • Do I have a quality user(s) relationship? • Do I make involvement easy? • Did I find out what the user(s) needs? For each question with a YES answer, add 3.8 points to the total project success potential score. Total Points (not to exceed 19) Executive Management Support • Do I have the key executive(s)? • Does the key executive have a stake in the outcome? • Is failure acceptable? • Do I have a well defined plan? • Does the project team have a stake? For each question with a YES answer, add 3.2 points to the total project success potential score. Total Points (not to exceed 16) Developing a Clear Statement of Reqts • Do I have a concise vision? • Do I have a functional analysis? • Do I have a risk assessment? • Do I have a business case? • Can I measure the project? For each question with a YES answer, add 3 points to the total project success potential score. Total Points (not to exceed 15) Proper Planning • Do I have a problem statement? • Do I have a solution statement? • Do I have the right people? • Do I have a firm specification? • Do I have attainable milestones? For each question with a YES answer, add 2.2 points to the total project success potential score. Total Points (not to exceed 11) Setting Realistic Expectations • Do I have clear specifications? • Do I have prioritization of needs? • Do I have small milestones? • Can I manage change? • Can I prototype? For each question with a YES answer, add 2 points to the total project success potential score. Total Points (not to exceed 10)

Small Project Milestones • Am I using the 80/20 rule? • Am I using a top-down design? • Am I setting time limits? • Am I using a prototype tool? • Can I measure progress? For each question with a YES answer, add 1.8 points to the total project success potential score. Total Points (not to exceed 9) Competent Staff • Do I know the skills required? • Do I have the right people? • Do I have a training program? • Do I have incentives? • Will the staff see it through? For each question with a YES answer, add 1.6 points to the total project success potential score. Total Points (not to exceed 8) Project Ownership: • Do I have defined roles? • Do I have a defined organization? • Does everyone know their role? • Are incentives attached to success? • Is everyone committed? For each question with a YES answer, add 1.2 points to the total project success potential score. Total Points (not to exceed 6) Clear Vision and Objectives • Is the vision shared? • Is the vision aligned with company goals? • Are the objectives achievable? • Are the objectives measurable? • Do I have honest sanity checks? For each question with a YES answer, add 0.6 points to the total project success potential score. Total Points (not to exceed 3) Hard Working, Focused Staff • Are there incentives? • Are we concentrating on quantifiable deliverables? • Does each member have part ownership? • Does everyone work together? • Are we building confidence? For each question with a YES answer, add 0.6 points to the total project success potential score. Total Points (not to exceed 3) Calculate all of the points to achieve the final score. The Success Potential for Project…

167

Appendix D: Pilot Study #1: Software Practitioner’s Success Factors – Survey Instrument The scale SA, A, N, D, SD refers to “Strongly Agree”, “Agree”, “Neutral”, “Disagree” and “Strongly Disagree”. For purposes of reference, you may assume that there is an equal ‘distance’ between each of the five possible responses. Section 1: Yourself and Your Organization 1.01 Please indicate your gender: Male

1.02

Female

Please indicate your highest level of education: Some high school High school/equivalent graduate Associate degree (2-degree) Bachelor degree (4-year) Doctorate (PhD)

1.03

Please indicate your primary job: Programmer Systems Analyst Network Engineer Database Administrator Team Leader Customer/User

1.04

Some college, no degree Graduate degree

Programming Analyst Project Manager/Leader Senior Manager

Other:

Please rate your IT/development/programming expertise: Expert

Proficient

Average

Somewhat inexperienced

1.05

How long have you been involved with IT? years

1.06

Please indicate your age range.

Quantitative models for early prediction of software ...

Quantitative models for early prediction of software ...

Suggest Documents

Early Quantitative Software Reliability Prediction Using Petri-nets

A technique for early prediction of software reliability ... - Springer Link

Case Study: Factors for Early Prediction of Software Success - CiteSeerX

Technique for Early Reliability Prediction of Software ... - PLOS

An approach for early prediction of software reliability

Software Quality Prediction Models Compared - CiteSeerX

Towards early software reliability prediction for computer forensic tools ...

Towards early software reliability prediction for computer forensic tools ...

A Novel Method for Early Software Quality Prediction ... - CUHK CSE

A Fuzzy Model for Early Software Fault Prediction ... - Semantic Scholar

Developing Software Bug Prediction Models Using Various Software ...

Software Defect Prediction Models for Quality Improvement: A ...

Hybrid Computational Models for Software Cost Prediction: An ...

Hybrid Computational Models for Software Cost Prediction: An ...

Early Warning? A Review of Conflict Prediction Models and Systems

A Brief Review of Software Reliability Prediction Models - IJRASET

Performance Prediction for Software Architectures

Development of probabilistic models for quantitative ...

Equivalence of Quantitative Models for Tumour ...

Chemometric Models For Quantitative Analysis of

Quantitative models for predicting adsorption of

Scoring Method for Early Prediction of Neonatal

The application of ANFIS prediction models for

3D NETWORK MODELS FOR PREDICTION OF ...