Software Project Risk in the Public Sector - CiteSeerX

Software Project Risk in the Public Sector Paul L Bannerman National ICT Australia Ltd [email protected]

ABSTRACT Controlling risk in software projects is a major contributor to successful outcomes, in both the private and public sectors. Few studies investigate risk management in the public sector. This paper reports a study of risk practices in government agencies in an Australian State. It finds that risk management practices lag behind project management practices and that there is scope for improvement in both. Ten major risk factors are identified. Also, some conventional conceptions of project and risk management are challenged. The paper concludes by discussing implications of the findings for research and practice in the public sector.

Keywords:

Software projects, risk management, project management, system development process, methodology.

1. INTRODUCTION Software projects are high risk activities, generating variable performance outcomes [7, 18]. International surveys suggest that only about a quarter of software projects succeed outright (that is, they complete as scheduled, budgeted and specified), and that billions of dollars are lost annually through project failures or projects that do not deliver promised benefits [9]. No consolidated data is available on software project performance in public sector agencies. However, evidence suggests that this is a global issue, impacting public and private sectors alike [18]. This paper has two aims. The first is to present the results of a study on risk practices in government agency software projects. The second is to draw implications from the study to identify how risk management might be improved. The paper is structured as follows. First we briefly review the notion of risk, as found in the literature, and identify some constraints in the dominant conceptions of risk. We then overview the study of public sector agencies and describe its main findings, including ten major risk factors. Implications are then drawn for research and practice in risk management. The main contributions of the paper are to focus attention on risk in public sector software projects; to challenge thinking about risk management and its relationship to project management, and; reaffirm the importance of management capabilities as well as engineering methods in risk research and practice.

1.1 Why is Risk Important? Risk arises when organizations pursue opportunities in the face of uncertainty, constrained by capability and cost. Consequently, risk management is a strategic and governance issue that usually involves compromise: a risk-averse strategy can limit distinctive achievement; however, a risk-embracing strategy can increase project failures.

Proceedings of the 2007 Australian Software Engineering Conference (ASWEC'07) 0-7695-2778-7/07 $20.00 © 2007

1.2 What is Risk? Risk in software projects is usually defined as the probability of a particular loss or impact on a project’s outcome [4]. That is, R = P x I, where R is the risk exposure attributable to a particular risk factor, P is the probability the risk will occur and I is the impact or magnitude of the loss of the unsatisfactory outcome. Risk exposure is usually measured in dollars or time. The general notion is that to reduce the likelihood of an adverse project outcome, all potential risk factors should be identified for the project. The risk exposure for each factor is then estimated (using the above formula) and the exposures are prioritized to identify the risks that represent the greatest threat to the project. Attention is then focused on the high risk factors to minimize the likelihood of them occurring (and/or the magnitude of impact if they do occur), through control measures such as mitigation strategies and/or contingency plans. Risk factors are monitored progressively to detect, as early as possible, when they come to fruition. Materialization of a risk factor is often recognized through the onset of a predefined risk trigger or the reaching of a predetermined risk threshold, at which time predefined contingency plans are activated to minimize the impact. The common definition of risk has some limitations. First, managers tend to be more concerned with the magnitude of the potential outcome than with the likelihood it will occur. Furthermore, they tend to prefer verbal characterizations of risk than probabilistic representations [12]. Second, it is very difficult, in practice, to estimate the probability of impact of many risk factors. Probabilities can only be meaningfully determined for activities that are repeated many times, under controlled circumstances. The one-off nature of many software project activities mitigates against this. A common response to this problem in practice is to view risk more generally in terms of uncertainty and to assess it qualitatively. Risk factors are assessed and ranked against a categorical scale of relative values such as low, medium and high on each of the dimensions of risk – likelihood and impact. Under this approach, ‘high-highs’ attract the most attention in applying risk control strategies, subject to cost. Third, the definition encompasses only known or foreseeable threats; it does not recognize unforeseeable threats (unknown unknowns). This limitation is usually resolved in practice through iterative risk identification rather than by specific processes in the risk methodologies for handling unforeseeable threats. In practice, ‘unknown-unknowns’ can materialize unexpectedly and rapidly, threatening the integrity of a project. Some organizations handle these threats through independent issue or crisis management processes (which are often managed less formally, and at a lower organizational level and priority than risks, and have less visibility to governance bodies). As a result, traditional risk management is

related to other important management processes such as issue management and crisis management, but this inter-relationship is rarely formally recognized in risk and project methodologies or explicitly accounted for.

1.3 What is Risk Management? Risk management is a set of principles and practices to identify, analyze and handle risk factors in software projects. Three common approaches are found: 1. Checklists. Checklists are lists of risk/success factors found in other projects. They can provide a quick, low cost (but rudimentary) way to manage risk by using them to check-off risk handling in a project. Many ‘top-ten’ lists are found in the literature [e.g., 5, 8]. However, there are a number of problems with this simple approach. First, how do you choose which list to use? There are many different lists available. Second, research shows that the perception of risk in software projects varies over time, between cultures, and between stakeholder groups, raising the likelihood that risk assessment based on published checklists might be biased and limited in scope [11, 20]. Third, research also shows that stakeholder groups tend to identify and rank highly risks that are perceived to be outside their own control. That is, they tend to identify risks in the responsibility domains of other stakeholders, rather than point to factors as risks within their own areas of responsibility [12, 20]. The effect of this can be to limit the likelihood that the full range of relevant risk exposures is identified, especially if the checklist is based on the perceptions of a single stakeholder group (which is usually the case with published lists). Therefore, we can conclude that (a) risk factor checklists are not universally applicable; (b) they are best used as a starter list for generating your own in-house list, and; (c) risk identification, analysis and review must include representative participation from all major stakeholder groups in the project. 2. Analytical Frameworks. Analytical frameworks provide broader ways of directing searches for possible risk factors. The most common type found in the literature is source categories of potential risks [for example, 2]. High level sources of risk such as technology, requirements, or expertise can each account for multiple risk factors. The categories can provide a broader framing for thinking about what risks might exist for a particular project, rather than to simply work through a pre-defined checklist of specific factors. The categories can also represent target areas for applying risk control strategies that might be effective on multiple factors. On the downside, many of the same limitations of checklists apply. Which framework should I use? Is it sufficiently representative of my project’s context? If not, should I use multiple frameworks, form a composite, or tailor a framework to my environment? And how much risk framing is enough to identify all relevant risks? The main limitation of this approach – as with any tool – is that its value depends on how well it is used. If the analysis is cursory or superficial then the risk management benefits are likely to be low. 3. Process Models. Process models specify stepwise methods for identifying, analyzing, and handling risks. Typically, these approaches specify the individual activities believed to be necessary to manage risk in software projects (for example, risk identification, analysis, response and control). Usually


they also specify how these activities should be sequenced to effectively manage risk and, less frequently, they may also suggest various tools and techniques to use in individual steps to aid in the risk management process. The ordered steps are usually intended to be executed iteratively throughout the project, to manage known and new risk factors as the project proceeds and as environmental circumstances change. Many prominent examples of risk management process models can be found in practice and in the literature. The two most dominant models in software engineering are associated with Boehm [5] and PMI’s PMBOK Guide [16]. Other influential models derive from Carnegie Mellon’s Software Engineering Institute (CMMI) and various industry and national standards bodies. A common weakness with process models is that they typically provide little guidance on how to implement them in practice. Also, being generic, they do not always fit individual project circumstances. While risk management can significantly improve software project performance, current approaches tend to be narrow in focus, understate management in favor of technique, and not always work well in practice [17].

2. THE STUDY The study investigated software project and risk management practices in government agencies in an Australian State. Access to agency projects was facilitated and coordinated by the government’s CIO office. Agencies were invited to participate in the study by nominating one or more recently completed software projects. Initially, 15 agencies nominated 21 projects. After issues of access and availability of informants was resolved, the study sample comprised 23 informant perspectives on 17 projects from 17 agencies. The correspondence, however, was not one project per agency. Ten agencies contributed one project each and three agencies contributed two projects each. Each project was different. One perspective was obtained on each project except for two projects on which two perspectives were obtained. A further project was investigated from the perspectives of four different, but related, agencies. Structured interviews were conducted with informants and a case study was prepared based on the informant’s perspective of the project. The interview comprised 150 questions spanning nine topic areas: informant, organization, project, governance, project management, development, implementation, third parties, and other. The questions were developed from a review of current research literature on project management and risk management, but also provided opportunity for informants to disclose contextspecific risk and success factors. Answers to questions reflected informants’ perceptions of the project. In most cases, participants were asked to answer by choosing a value from one to ten, where one signified ‘no’ or the lowest value in response to the question, and ten indicated ‘yes’ or the highest value. They were then given the opportunity to explain their answer. For example, “Was the project completed within budget?” might have been answered with a value of eight and the explanation that a slight cost overrun occurred but it was within limits considered acceptable by the informant. Interviews lasted from one to two hours and were recorded. Interview recordings were later used by the researcher to generate

descriptive statistics and prepare a descriptive case study of the informant’s account of the project. Each case study was analyzed to identify thematic patterns, apparent constructs and/or possible relationships. Over 300 artifacts were identified by multiple researchers as evidentiary data that appeared to be relevant or important in the performance and/or outcome of the project. Interest focused on novel characteristics and events that directly related to the projects studied and their contexts, rather than on aspects of project and risk management already widely found and discussed in the software engineering literature. The artifacts were then sequenced by theme, resulting in ten categories of software project risk factors. Further analysis of these categories gave rise to novel insights into software project risk in the cases studied. While not random, the study sample comprised considerable variety in agencies and projects, which are profiled following.

2.1 Agency Profile The state government agencies in the study varied significantly in size, ranging from very small (31 employees) to very large (130,000 employees). The average size was 9,800 employees, but this figure is biased by one very large agency. The median size was 650. The majority of agencies comprised less than 5,000 employees, as shown in Table 1. Table 1: Agency size Size (in employees) Less than 1,000 1,000 – 5,000 5,000 – 10,000 10,000 – 100,000 More than 100,000

Percentage of agencies (%) 50 38 6 0 6

The reporting level of the CIO (or CIO equivalent) is shown in Table 2. This is an indicator of the relative importance and role of the IT function in the agency. Typically, IT reported to a divisional or functional director, such as Corporate Services. Table 2: CIO reporting level CIO reporting level CEO (or equivalent) Divisional Director Business Unit Manager

Percentage of agencies (%) 38 56 6

Informants were mostly project managers, as shown in Table 3. Table 3: Study informants Interviewees Project Manager IT Manager Business Manager

Percentage (%) 70 22 8

For the informants that had project management experience, the length of experience was typically more than five years. The average duration of experience was 5 to 10 years, but there was some variation in experience, as shown in Table 4.


Table 4: Project management experience Project management experience Less than 2 years 2 – 5 years 5 – 10 years More than 10 years

Percentage (%) 12 18 35 35

2.2 Project Profile More than half of the projects in the study involved development and implementation of software (see Table 5). Another one-third involved implementation only. Table 5: Project type Project type Development and implementation Implementation only Development only Other

Percentage (%) 53 32 10 5

Web applications and transaction-based systems development and implementation dominated, as summarized in Table 6. Table 6: Applications Applications Web-based Transaction-based Package implementations Data publishing Statewide package rollout

Percentage (%) 35 29 24 6 6

More than half of the projects in the study were less than 12 months long in duration on completion (see Table 7). Three were large and ongoing, in which case interviews focused on the completed phases of the projects. Table 7: Project size Project size < 6 months 6 – 12 months 1 – 2 years > 2 years

Percentage (%) 22 34 22 22

Most projects involved and were significantly dependent upon one or more external third parties (vendor, developer, consultant, etc.). The number of participating third parties ranged from zero to three, with an average of two per project. Finally, 89% of projects were perceived as having some importance in achieving the strategic objectives of the agency, with two-thirds (67%) seen as being highly relevant, strategically. Only 11% of projects were seen as not being of strategic importance to the agency. In contrast, however, only one third (33%) of the projects had CEO (or equivalent) level involvement. In another half of the projects (50%), the most senior executive involved was a divisional or functional director; and in 17% of the projects, the most senior person involved was a business unit manager.

Table 9: Project governance

3. FINDINGS 3.1 Responses on Some Key Measures Project outcome. Table 8 summarizes responses to questions relating to project outcome (in Tables 8-13, following, ‘Score’ is the average score and ‘Range’ is the spread of scores from 1-10). All of the projects in the study were perceived to have been ‘successful’ except for one, which was suspended indefinitely due to conflicts with contingent projects (and, therefore, is not included in these responses). Informants’ claims of project success were accepted at face value. Before being asked the questions in Table 8, participants were probed to find out how they define ‘project success’. Responses varied, indicating multiple views on what constitutes a ‘successful’ project. Predictably, business executives and managers tended to answer in terms of achievement of business objectives, although one response was in terms of achievement of project objectives (on time, within budget, etc.). The following are examples of business executive/management conceptions of project success: • • • •

extent of achievement of required outcomes the business benefits were actually derived achievement of project goals/objectives and provision of identifiable business efficiencies full deployment on time and within budget

•

internet query response time within 6 seconds having a system that works as expected meets objectives on time, within budget and resources, and satisfies customers meets the business drivers (whatever they are)

Table 8: Project outcome Questions How successful was the project? Were the business objectives achieved? Was the project completed within budget? Was the project completed within schedule? Was the project completed within scope? Were the objectives clearly defined? Did the objectives change during the project?

No A little Moderately Substantially

Was the scope clearly defined? Did the scope change during the project?

No A little Moderately Substantially

Was an appropriate budget set? Was a realistic schedule defined? Was the project adequately resourced?

Score 8 8.9 9 8.4 8.8 8.4 74% 16% 5% 5% 8.9 50% 30% 15% 5% 8.6 7.5 8.3

Score 75% 25% How effective was the steering committee? 7.9 Was there a project sponsor? Yes 94% No 6% How effective was the project sponsor? 8 How committed was top management? 9.2 How involved was top management? 7.7 Did this commitment/involvement benefit the project? 7.9 Did the project team feel they had executive support? 8.6 What level was the most senior executive directly 33% involved in the project? CEO Director 50% Business unit manager 17%

Range

Yes No

3-10 1-10 4-10 3-10 1-10 4-10

Project management. Ratings on project management (Table 10) tended to be lower than those for project outcome (Table 8), and displayed greater variance. Note also that the question on whether a formal PIR was held attracted the lowest score so far, with high variance. Table 10: Project management

In contrast, responses by IT and project managers varied in emphasizing technical considerations, achievement of project objectives, and achievement of business objectives and soughtafter organizational outcomes. For example: • • •

Questions Did the project report to a steering committee?

Range 5-9 7-10 6-10 7-10 6-10 3-10

5-10

6-10 3-10 5-10

Project governance. Table 9 summarizes responses relating to project governance. Three quarters of all projects reported to a steering committee and nearly all had a business sponsor. However, only a third of projects had direct CEO involvement.


Questions Score Did the project have a full-time project manager? Yes 81% No 19% How well was the project managed? 7.8 Did the project mgr have a clear vision of the project? 7 Did the project mgr change during the project? Yes 28% No 72% How effective was project planning? 7.8 How appropriate was the methodology used? 7.4 How well was progress controlled against the plan? 7.1 Did you know the project’s status most of the time? 7.6 How well were issues handled during the project? 8 Did the project have effective change control? 8.2 Was a formal post-project review held? 6.5

Range 6-9 4-10 4-10 1-10 2-10 6-10 4-10 2-10 1-10

Risk management. Average scores on risk management practices dipped in value even further, with all questions attracting responses across the full range of scores (Table 11). The responses indicate that agencies in the study tended not to use a specific/formal risk management methodology, and were quite equivocal about the prioritization of risks. These responses are curious considering that a majority of projects did encounter unanticipated threats and they were considered to be strategically important (Table 12), indicating the importance of having strong risk management. No agency reported using quantitative risk assessment. Agencies that assessed risk used qualitative scales. Table 11: Risk management Questions Did you use a specific risk management methodology? How well were risks identified as the project’s start? How well were risks managed throughout the project? How well were risks prioritized when identified? Were mitigation/contingency plans pre-determined? Was responsibility assigned for monitoring risks? Did unanticipated problems arise during the project?

Score 4.6 6.9 6.6 5.3 7.6 6.4 7.7

Range 1-10 1-10 1-10 1-10 1-10 1-10 1-10

Development. Insufficient responses were received on questions relating to software development practices in the projects studied. The three most common reasons were that: (1) the informant did not feel sufficiently informed to answer; (2) the software was

developed by a third party, and/or; (3) the project did not involve software development. Implementation. Informants scored the success of system implementation quite highly (Table 12), which is consistent with the perceptions on project outcome (Table 8). There was also strong consistency between perceptions of the scope of impact of the system being implemented on the organization and the project’s relevance to achieving strategic objectives of the agency. Informants scored the projects moderately high on how well organizational and staff changes associated with the system implementation were managed (for example, process realignment, staff training, user support, etc.), but few projects explicitly included organizational change management within their scope. In most cases, any related organizational change management was handled independently by the business areas impacted by the new system, either proactively or reactively as the system was introduced. Table 12: Implementation Questions How easy was implementation? (0 = trivial) How successful was the implementation? How great was the impact on the organization? How important was the project to strategic objectives? How well were the organizational changes managed?

Score 6.5 8.4 7.1 7.9 6.9

Range 2-10 4-10 2-10 2-10 4-9

Third parties. Finally, as noted earlier, most projects had third party (‘vendor’) involvement. Overall, their input was considered to be effective, with some variation, and the relationships were perceived as having been well-managed (Table 13). Table 13: ‘Vendor’ management Questions How effective was third party input to the project? How well were third party relationships managed??

Score 7.5 8.1

Range 2-10 5-10

3.2 Project Types We tend to think of software projects in simple terms as discrete activities subject to the joint disciplines of project management and software engineering. However, four distinct project types were identified in the study along two dimensions: function (the degree of organizational relatedness) and form (the degree of relatedness to project disciplines). The four types are the ‘pure’ project form, operational activity, hybrid form and breakthrough event (Figure 1). PMI’s PMBOK Guide [16] distinguishes between projects and operational work on the basis that the former is a discrete activity and the latter is ongoing. However, the study indicated a more complex reality in the public sector. More importantly, the characteristics of the types suggest that different issues and challenges arise with respect to project and risk management in each quadrant, as summarized in Table 14. (Note that these types relate only to internal activities; they exclude consideration of additional variations made possible through external party involvement.)


Figure 1: Project Types ‘Pure’ Project form. This is the traditional type of project – a discrete, temporary activity that is structured, operated and managed under conventional project management disciplines (to varying degrees of formality and completeness). In the public sector, its key advantage is that it enables dedication of effort to the target objectives, and its main weakness is in getting buy-in from stakeholders outside of the project team. Its major challenges are the adequacy of project and risk management skills, as well as achieving and maintaining organizational alignment. Operational activity. This is a recurring activity that is conceptualized as a project but is executed by functional units within their normal operational space and structure, using elements of project management control (like schedule, budget, reporting, etc). Its key advantage is that it avoids disruption to functional business units and artificial overheads, and its main weakness is that project and risk management disciplines are subordinated to business unit priorities and practices. Its main challenge is in balancing operational practices and expedience with accountability for delivery and quality control. Hybrid form. This is a combination of the ‘pure’ project form and operational activity. A core project structure exists, but key elements of the project are delivered by one or more functional units that exist and operate independently of the project. It has the advantage of optimizing the use of specialist resources, but the disadvantage of being difficult to enforce compliance and accountability. Its main challenge is in balancing competing objectives, practices and resource demands. Breakthrough event. This last type is a focused effort by a small dedicated team to achieve a specific, high priority objective in a short timeframe without the constraints of conventional disciplines, practices or methods. Its main advantage is that it focuses effort on quick results, but is highly dependent on ‘heroes’ (the skills and dedication of individual team members). Its key challenge is that it tends to leave ‘loose ends’, ill-fitting solutions and unresolved issues that hamper subsequent operations. The direct implication of this finding is that a uniform view of, and approach to project and risk management is unlikely to fully address the specific challenges associated with all project types. That is, ‘one size’ project and risk management does not ‘fit all’.

Table 14: Types of Projects ‘Pure’ Project Form

Hybrid Form

Operational Activity

Breakthrough Event

Characteristics

Traditional project. A discrete, temporary activity, structured, operated and managed under conventional project management disciplines (to varying degrees of formality and completeness).

A combination of project form and operational activity. A core project structure exists, but key elements of the project are delivered by one or more functional units that exist and operate independently of the project.

A recurring activity that is conceptualized as a project but is executed by functional units within their normal operational space and structure, using elements of project management control (like schedule, budget, reporting, etc).

A focused effort by a small designated team to achieve a specific, high priority objective in a short timeframe without the constraints of conventional disciplines, practices or methods.

Project mgmt

Formal

Semi-formal

Informal

None

Locus of control

Project manager; governance framework

Project and functional management

Functional and line management

Executive

Time horizon

Variable

Short-medium

Short – medium

Short

Advantages

Enables dedication of effort to target objective.

Optimizes use of specialist resources.

Avoids disruption and artificial overheads.

Focuses effort on quick results.

Weaknesses

Getting buy-in and participation from stakeholders outside of the project team.

Difficult to enforce compliance and accountability.

Project/risk management disciplines subordinated to business unit management/practices.

Highly dependent on ‘heroes’ (the skills and dedication of individual team members).

Key challenges

Adequacy of project and risk management skills; achieving/maintaining project & organizational alignment.

Balancing competing objectives, practices and resource demands.

Balancing operational practices and expedience with accountability for delivery and quality control.

Tends to leave ‘loose ends’, ill-fitting solutions and unresolved issues that hamper subsequent activities/operations.

3.3 Major Risk Factors Analysis of the projects in the study uncovered ten clusters of related risk factors that tended to be common to the participating agencies. These are summarized in Table 15. Table 15: Major risk factors Risk Factor categories Effective project governance framework Realistic project setup Value-adding partner engagement Strong business proprietorship Experience in project management Manage organisational change concurrently Recognition that projects are a management activity Ability to recognise red flags Recognition that risk management is more than a process Recognition that benefits must be sought and captured

The categories are not ordered by importance or priority. The study did not distinguish their relative importance; they are all important in reducing software project risk. Instead, the categories are loosely sequenced according to their positioning in the software development life cycle (SDLC). The ten categories are briefly described following.


Effective project governance framework. A range of governancerelated issues were found in the projects studied. Projects tended to struggle if governance bodies were passive, have insufficient or inadequate representation, mixed or declining participation rates, or did not actively engage and drive the project as a business or organizational initiative. Effective project governance was found to facilitate business alignment, executive involvement, business ownership, clear objectives, scope and requirements; and provide guidance, direction and a common sense of purpose. Effective governance did not curb the project manager/team’s responsibilities or stifle initiative by the project. An absence of effective governance resulted in risk exposures in these areas. Two forms of effective project governance were found. The most common form was the project steering committee (PSC), chaired by a committed and involved senior business executive. In other projects, there was no formal PSC (or the PSC adopted a very passive role) so ‘governance’ was effectively provided through the proprietorship of a business owner or project sponsor at the project level. In these cases, the business person and project manager formed a close operating relationship, based on mutual dependence, and worked through project issues together as they arose. Executives were informed only on major issues with organizational impacts, issues that might affect achievement of project objectives, and progress against key milestones. An effective model in dynamic project contexts was found to be a framework that permitted flexibility in day-to-day decisionmaking at the project level, in consultation with business owners,

and the ability to escalate major issues quickly, to draw on broader executive input and support. Realistic project setup. Many projects encountered issues (risks) due to poor project setup. Determining the best approach, the most efficient project design and development methodology, setting the right budget, and securing the necessary funds were found to be critical activities to get right at the start of a software project. Problems tended to arise from using rigid, plan-based methodologies when requirements were uncertain and/or contexts were volatile, and being constrained by funding arrangements that allocated project funds before project costs were fully known. Other issues arose from project setups that did not align to valueadding business objectives or where project setup was devolved to a dominant vendor whose priorities and actions were driven by self-interest. Value-adding partner engagement. Several projects found that external third parties can be an asset to the project or a significant risk. The key challenges were in engagement (outsourcing versus insourcing) and control (retaining or surrendering responsibility for the project). For engagement, an ‘arms length’ outsourcing approach was found to work well in formally structured, plan- and specification-driven contexts, but lacked the flexibility and interaction necessary under conditions of greater uncertainty. Some, especially smaller, agencies learned over time that better value and control could be achieved by having the third party developer’s project staff located in-house (insourcing). This physical integration greatly improved project communication, interaction, issue resolution and progress tracking. It also enabled a degree of incremental, cyclical development, which resulted in delivery of a system that more closely fitted the agency’s needs. With respect to control, some agencies did not seem to recognize the risks in not maintaining project control. For example, rather than prepare their own plan and use their own methodologies, some agencies defaulted to using whatever the dominant third party proposed or used, placing themselves fully in the hands of their project partner(s), thereby surrendering control. In another case, great effort was taken to outsource the technical risk associated with a project to the vendor via the contract. This appears to have been done effectively as technical risk did not impact the project (although we do not know if the foreshadowed risk materialized or not). However, the agency’s view was that it had given the risk away and no longer ‘owned’ it. This is dangerous (risky) thinking. Ultimately, organizations cannot offload software project risk. They can outsource risk management responsibility for certain risk factors and, perhaps, even offset the cost of the impact if the risk materializes (through penalty clauses), but if the required outcome is not fully delivered, then it is the client organization that takes the direct impact of any sunk costs not covered by the penalties, as well as opportunity loss and cost. Other agencies, however, found that rather than being a source of threat to the project, the prime contractor contributed positively to the project and its outcomes through the experience and competence of their project/interface manager and specialist staff. Finally, managing input from internal third parties was also a problem for some projects. Some relied on the IT department, for example, to provide infrastructure or implementation services to the project (in hybrid form projects). Conflicting priorities often made these unreliable partners. To overcome this problem, one


agency planned to implement a project-based service level agreement (SLA) during project initiation, as a formal engagement of internal service providers. Strong business proprietorship. Business ownership, sponsorship and participation were found to be critical project risk/success factors. In fact, several project managers claimed that committed and capable business sponsorship was the most important factor in the success of their projects. Two effective approaches were found: formal proprietorship by the business sponsor through project governance structures and; informal relationship through the business owner teaming with the project manager (as discussed above). Both could be effective. Strong business sponsorship was found to remove obstacles and drive projects toward their objectives. Projects with weak or no sponsorship tended to face escalating project issues. Experience in project management. Project management capability is a critical risk/success factor. However, at least one project found that while good project management is necessary, it is not enough for success. Organizational capabilities were also needed to support and enable individual project managers and projects. These include development programs, mentoring arrangements, knowledge databases, career structures and incentive schemes to deliberately promote, build, reward and support in-house skills in project activities. Agencies that recognized the need to grow internal capabilities in project management were found to build mechanisms into their project governance frameworks – typically through a project office – to capture project experience and knowledge and pass it on to others for the benefit of future projects. Agencies that relied on project managers alone, tended to be exposed to the limitations and vagaries of project ‘heroes in action’. Manage organizational change concurrently. Many projects in the study encountered implementation and user-related issues due to inadequate management of organizational impacts. Three issues were found: (1) sometimes software solutions were imposed on the business; (2) business processes were not always realigned to the new applications; and/or (3) user buy-in and resistance to change were not managed. Projects tended to have fewer implementation problems when organizational change was managed concurrently from the beginning of the project. In these cases, software projects were viewed as ICT-enabled organizational change events. That is, as a technical means to a business end. Recognition that projects are a management activity. Some paradoxical observations of the projects studied raised questions about the traditional view of project management as a formal discipline of defined methods and practices that are critical for project success. For example, one project used no formal project management methodology or practices whatsoever but succeeded, and another (mentioned above) rigorously followed project management ‘by-the-book’ but essentially failed. In contrast to the earlier view that project management is necessary but not sufficient for success, these observations challenge whether formal project management, as traditionally conceived, is even necessary for project success. What these projects indicate is that project success is an outcome of good management, per se, not ‘project management’ as a body of knowledge and practice. This view is consistent with Morris’s [13] distinction between the management of projects and project

management. That is, there is no inherent magic in the bundled software engineering principles and practices called ‘project management’ that guarantees success. What is critical to software project success is good management of a discrete, temporary, joint business-technical activity. Let’s be clear, this is not a call to abandon formal disciplines of project management in favor of an “anything goes” approach. History shows that formal ‘project management’ can be a very effective way of shaping how we manage software projects. But it is no panacea. Rather, it is a call to embrace software projects within the organization’s management structures and practices, whether through formal methodologies or otherwise, rather than naive submission to a packaged ‘project management’ solution. This finding supports the view that we need to build individual and organizational capabilities in managing projects, not just in formularized methods. Ability to recognize red flags. The study found that certain characteristics of individual projects tend to increase risk. The ability to recognize risk indicators, take them into consideration in building and deciding the business case, and designing, planning and executing the project, was found to be critical to good risk management and, ultimately, the outcomes of the project. Risk indicators found in the study include: • • • • • • • • • • • • • • • • • • •

Enterprise-wide application implementation No business or executive involvement Organizational/business objectives undefined or unclear Data conversion or migration New/unfamiliar technology, methodology, tools, techniques No formal project plan or methodology Inexperience or lack of skills in project team Resource limitations Lock-stepped critical path or rigid deadline Technical performance/throughput critical Other significant organization-specific project constraints Key stakeholder with contrary or competing objectives No representative business sponsor/user External or internal dependencies, especially multiple Concurrent interdependent or contingent projects Vendor involvement Large end-user base External funding Changed organizational/contextual circumstances

These characteristics were the source of various project-specific risks and a major reason why generic top-ten checklists may not focus attention on all relevant threats to an individual software project. A critical competency for organizations to build is the ability to recognize red flags like these and thoroughly assess and treat the associated risk exposures to the project. Recognition that risk management is more than a process. The study suggests that risk management, like project management, is more than a methodology; it is also a real-time threat management capability that is developed within an organization, through learning and other mechanisms, over a long period of time. Risk management is not just about identifying risks, and putting in place mitigation and contingency strategies. It is also about being able to respond quickly and effectively to realized project threats as they arise.


This was graphically illustrated by one agency that was exposed to an extreme risk of security invasion. The agency thoroughly planned and prepared for the threat, taking every reasonable mitigation action, including predefining contingency plans and responses in the event a breach occurred. A security breach did occur, but in an unexpected way. To the uninformed observer, this may have looked like a failure of risk management. However, this is an unreasonable conclusion because it is highly unlikely that every possible threat could have been foreseen and catered for. In this case, the breach was sealed off within minutes of it being identified, with no damage to the system or its integrity. This incident demonstrated an effective and legitimate form of risk management to contain a risk that was not identifiable until after it materialized. The response to the threat was an effective utilization of an organizational risk management capability – that has been built up over many years – being rapidly applied to contain the impact and eliminate the exposure. This capability dimension of risk management is essential to handle unforeseen threats that suddenly materialize and cannot be responded to through planning-based approaches. Recognition that benefits must be sought and captured. Finally, the cases studied also indicated that it is unlikely that benefits will flow automatically from software projects. At the business level, ICT projects carry the cost, but business efficiencies generate the benefits. Therefore, benefits have to be both sought (such as during project setup) and captured within the business-ICT collaboration and project governance mechanisms. Delivering a project “on time, within budget and to specification” is of limited value if the original driving goals of the project are not also actively sought and achieved. In the projects studied, for example, one agency realized, at the end of a major project, that little more had been achieved than to install a new ICT-based capability – a new system – but no real organizational benefits had been achieved. Therefore, the agency planned another project cycle to focus on realigning processes, deploy new functionality available in the system, and consolidate its integration with other critical enterprise systems. The driver for this approach was a change of mindset that occurred within the PSC from viewing the new application as an administrative system to a value-adding operations support system. At the project level, seeking and capturing benefits also includes securing returns to future software projects, not just organizationlevel stakeholders. One agency held PIRs at the end of each software release, in a multi-phase/release project, to ensure that the incremental benefits were being delivered and that lessons learned were carried forward to the next release. These mindsets ultimately deliver real value to the agency from the IT investment. In other cases, however, the same mindset or pursuit of benefits realization was not so evident. There were cases, for example, where a new system generated limited value because insufficient data had been loaded or migrated. In another case, it was noted at the end of a project that benefits realization was not yet evident – but neither was it evident that there were any mechanisms in place or momentum to seek benefits. Finally, in one other case, there was no known business case or statement of the expected benefits for the project or organization to target.

4. DISCUSSION This paper has examined project and risk management practices and challenges in 17 software projects in government agencies in an Australian State. The overall aim of the study was to contribute to a gap in research on risk management in the public sector. The study confirms that much of current ‘best practice’ project and risk management is equally relevant to the public sector as it is to the private, but it also challenges some traditional thinking about software projects and risk. The study is preliminary and conclusions can only be drawn tentatively, not the least because most projects were viewed as successful (which leaves less room to argue for improvements). However, the study made no attempt to validate these perceptions from the agencies’ perspective or against any objective benchmark of project performance (from the researchers’ perspective, the success of some of the projects was equivocal).

4.1 Summary Findings In summary, three findings of the study are highlighted. First, against the benchmark of current industry and government sector ‘best practice’, risk management practices appear to lag project management practices in the public agencies studied. Risk management tended to be unsystematic or informal. Risk was not well understood or well managed in many agencies. This is surprising considering the high level of scrutiny and accountability that operates in the public sector in Australia. To further improve risk-based outcomes from software projects, there is a need to develop greater awareness of the nature of risk and risk management and the potential to improve project outcomes, and evolve better practices [15]. These are inextricably linked with project management. Second, the study suggests that many of the risks that threaten software projects are built into the project design at the start of the project life cycle. For example, rigid plan-based waterfall development and structured project management methodologies dominated the projects studies. However, use of this design in uncertain and changing environments can become a major source of system and project risk in itself. One option to avoid this is to match the project and risk management approaches used to the characteristics and contexts of each software project (that is, adopt a contingency approach to design) [3, 21]. Another is to apply adaptive learning [22]. However, few guidelines are available to aid organizations in this process (a notable exception is [7]). Third, the study findings suggest that agencies tended to expect too much from engineering solutions to management problems. There was a tendency to undervalue and under-employ critical organizational management capabilities to enable and support software projects in favor of an unquestioning acceptance of inflexible, formularized and often generic methods. This (again) is curious considering the (sometimes) large investments being made in ICT in the sector as enablers of agencies ability to strategically fulfill their charters of operations. For example, the rapid onset of unforeseen threats and disruptive changes in an agency’s or project’s context require rapid response management borne of deeply embedded capabilities in adaptation, improvisation and value creation [14]. These management capabilities are developed in-house over long periods of time by deliberate learning from experience and institutionalizing


accumulated project and risk management competencies into the structures, processes and practices of the organization [1]. This cycle of development also reinforces progressive refinement of project and risk management methods as critical process components of an organization’s capability [19].

4.2 Research Implications One of the objectives of the study was to expand the current research base on software project risk by examining an industry sector that invests large amounts of money in software-based systems but has been poorly represented in prior studies. The study contributes a set of public sector software project risk factors that is different to most current lists. While these factors are mostly consistent with current research understandings of risk and risk management, the study offers specific insights in shaping a foundational theory of risk management. In particular, the study supports calls by other researchers, from both within and outside the software project domain, for a contingency-based approach to project management [3, 21]. This study provides an additional motivation for this approach as a critical driver of risk control. The study also reaffirms management as a fundamental anchor in project and risk research. The push by commercial interests for packaged project/risk solutions must not distract researchers from grounded investigation of software projects as organizational change initiatives that need to be managed as business investments as well as engineering activities. In their promotion of ‘project management’ we must not loose sight of the need to build organizational capabilities in the ‘management of projects’. Engineering-based methods and techniques are valid tools for managers but are of limited value in unskilled hands, used naively, or used as ends in themselves. Project management and risk management initially developed from management and organization theory. Project and risk research needs to continue to build a theoretical framing of project/risk as a managerial pursuit.

4.3 Practice Implications Many implications for agency practitioners arise from the study. Three in particular are highlighted. First, there is a need to redress the current imbalance in the role of risk management in software projects. Agency feedback suggests this has more to do with an underdeveloped appreciation of its role and value than a view that it is not needed or beneficial. Second, the typical structuring of IT as a second tier function in agencies increases the need for project governance – whether formal or informal – to provide the critical IT-business linkages required to make risk management effective at the enterprise level. Third, early life cycle activities such as project planning and design have unique challenges for the public sector. Generic project and risk management methodologies need to be adapted to take these into account. Finally, one of the key challenges in improving risk management in practice is that if the project is successful, it can be difficult to unequivocally attribute that outcome to risk management. This is exacerbated by the tendency for project success to be claimed by stakeholders as resulting from their contribution to the project. On this basis, it can become easy for an organization that has had a success to play down the importance of risk management in the next project. This can happen implicitly by being less formal in executing risk management processes next time or explicitly as

some form of cost rationalization. One way to mitigate this risk is to ensure that assessment of the role and effectiveness of risk management is included in post-project reviews.

4.4 Limitations The findings need to be interpreted cautiously because the study has some limitations. First, the data sample is small, not random and dominated by cases that were perceived to be successful. This may have limited or biased the findings of the study. Second, the risk factors described in the paper flow from the findings of the study and therefore have high ‘face validity’, but they have not otherwise been empirically validated. Third, performance outcomes of the projects studied were based solely on the perceptions and reports of the informants. More objective measures are needed in future studies to provide a firmer foundation for drawing implications from the findings. Finally, the study has focused only on software projects; other project types such as infrastructure, process redesign and outsourcing projects are also common in government agencies.

4.5 Further Research Areas for future research include: extending and refining the study by examining a wider, more balanced range of projects spanning all types (software, infrastructure and BPR projects), sizes (small, medium and large) and outcomes (successes and failures); investigating the project and risk management implications of the four types of projects found in the study, and; researching and developing a contingency approach to project and risk management, based on an integrated process model of software project formation and execution.

5. CONCLUSION This study has contributed insights into risk and project management practices and challenges in public sector agencies. Software projects are complex multi-dimensional endeavors in any context, private or public. While prescriptions and practices are well-developed in project management, agencies examined in this study suggest that risk management is underdeveloped in software projects and not well integrated in project methodologies. The cost and potential losses from software projects is high. Researchers and practitioners must continue to learn from each other to reduce project failures and develop practices that consistently generate better project outcomes. Better risk management, as an organizational capability, is critical to achieving these objectives.

6. REFERENCES [1] Bannerman, P.L. The Liability of Newness: Toward a Capability-based Theory of Information Systems Performance. PhD Dissertation, University of New South Wales, Australian Graduate School of Management, 2004. [2] Barki, H., Rivard, S. & Talbot, J. Toward an Assessment of Software Development Risk. Journal of Management Information Systems, 10(2), 1993, 203ff. [3] Barki, H., Rivard, S. & Talbot, J. An Integrative Contingency Model of Software Project Risk Management, Journal of Management Information Systems, 17( )4, 2001, 37-69.


[4] Boehm, B.W., Software Risk Management, Tutorial, IEEE Computer Society, Washington, 1989. [5] Boehm, B.W. Software Risk Management: Principles and Practices. IEEE Software, 8(1), 1991, 32-41. [6] Boehm, B. & Turner, R. Using Risk to Balance Agile and Plan-Driven Methods. Computer, 36(6), 2003, 57-66. [7] Boehm, B. & Turner, R. Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley, 2004. [8] Johnson, J., Boucher, K.D., Connors, Y. & Robinson, J. Project Management: The Criteria for Success, Software Magazine, 21(1), 2001, S3-S11. [9] Johnson, J. My Life is Failure, Standish Group, 2006. [10] Keil, M., Cule, P.E., Lyytinen, K. & Schmidt, R.C. A Framework for Identifying Software Project Risks. Communications of the ACM, 41(11), 1998, 76-83. [11] Keil, M., Tiwana, A. & Bush, A. Reconciling User and Project Manager Perceptions of IT Project Risk. Information Systems Journal, 12(6), 2002, 103-119. [12] March, J.G. & Shapira, Z. Managerial Perspectives on Risk and Risk Taking. Management Science, 33(11), 1987, 14041418. [13] Morris, P.W.G. Project Management: Lessons from IT and Non-IT Projects, in Earl, M.J., (Ed.). Information Management: The Organizational Dimension, Oxford University Press, Oxford, 1996, 321-336. [14] Pavlak, A. Project Troubleshooting: Tiger Teams for Reactive Risk Management, Project Management Journal, 35(4), 2004, 5-14. [15] Pfleeger, S.L. Risky Business: What We Have Yet to Learn about Risk Management, Journal of Systems and Software, 53(3), 2000, 265-273. [16] PMI. A Guide to the Project Management Body of Knowledge (PMBOK Guide). Project Management Institute, 2004. [17] Ropponen, J. & Lyytinen, K. Can Software Risk Management Improve System Development: An Exploratory Study, European Journal of Information Systems, 6(1), 1997, 41-50. [18] Sauer, C. & Cuthbertson, C. The State of IT Project Management in the UK. ComputerWeekly.com, 2003, http://www.cw360ms.com/pmsurveyresults/index.asp. [19] Sauer, C, Liu, L, Johnston, K. Where Project Managers are Kings, Project Management Journal, 32(4), 2001, 39-49. [20] Schmidt, R., Lyytinen, K., Keil, M. & Cule, P. Identifying Software Project Risks. Journal of Management Information Systems, 17(4), 2001, 5-36. [21] Shenhar, A.J. One Size Does Not Fit All Projects: Exploring Classical Contingency Domains, Management Science, 47(3), 2001, 394-414. [22] Sommer, S.C. & Lock, C.H. Selectionism and Learning in Projects with Complexity and Unforeseeable Uncertainty, Management Science, 50(10), 2004, 1334-1347.