Modeling software bidding risks - Software Engineering ... - IEEE Xplore

542

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 29,

NO. 6,

JUNE 2003

Modeling Software Bidding Risks Barbara Kitchenham, Member, IEEE Computer Society, Lesley M. Pickard, Stephen Linkman, and Peter W. Jones Abstract—We discuss a method of developing a software bidding model that allows users to visualize the uncertainty involved in pricing decisions and make appropriate bid/no bid decisions. We present a generic bidding model developed using the modeling method. The model elements were identified after a review of bidding research in software and other industries. We describe the method we developed to validate our model and report the main results of our model validation, including the results of applying the model to four bidding scenarios. Index Terms—Software projects, bidding, risk, software cost estimation, Monte Carlo simulation, software bidding model, model validation, contingency costs.

æ 1

INTRODUCTION

A

LTHOUGH this paper is concerned with the bidding for software projects, its origins lie in the field of software cost estimation. We have, for many years, been actively involved in the area of software cost estimation. We have presented the basic concepts of cost estimation, such as collecting data about past projects and use of that data to build (or calibrate) predictive models, to many software practitioners. Although they agree in principle with our recommendations, they usually have a number of reservations:

Discussions with practitioners have convinced us that early estimates are both the most difficult to obtain and the most important. However, we need to cope with the uncertainty inherent in estimates and to incorporate estimation into a defined management process. For this reason, we have investigated the software bidding process and have developed a generic bidding model that caters for uncertain estimates and allows managers to visualize the risk inherent in the bidding process. The goals of our model are: To link the model to the software bidding process and collate information available from the different organization roles. . To illustrate how uncertainty in estimates affects bidding decisions. . To address the concerns of management about fixing an appropriate price, understanding the risks involved in the proposal, and how to manage them. First, we describe the requirements we placed on our bidding model and how they were derived. Next, we present the generic software bidding model that we developed to meet these requirements. We then discuss our model validation exercise describing the evaluation process we developed and the results of applying that process. .

In our courses, we point out that estimates are essentially inaccurate. Following DeMarco’s ideas, we recommend estimating upper and lower bounds, not just a single point estimate [6]. However, in discussion with senior managers and estimators, we found that they want a fixed price, not an uncertain estimate. Managers and estimators did not know how to respond to estimate uncertainty. It was regarded as an unnecessary complication. . Practitioners point out that early estimates are extremely inaccurate and that data intensive methods only provide good estimates when it is too late to use them constructively. They want better initial estimates to improve bidding activities and budget setting, not measures that tell them late in the development process that their plans are infeasible. In addition, we ourselves have been guilty of forgetting that cost estimation and other metrics activities are not ends in themselves. They need to be integrated into management and development processes before they will be adopted. .

. B.A. Kitchenham, L.M. Pickard, and S.G. Linkman are with the Computer Science Department, Keele University, Keele, Staffordshire, ST5 5BG, UK. E-mail: {barbara, lesley, steve}@cs.keele.ac.uk. . P.W. Jones is with the Mathematics Department, Keele University, Keele, Staffordshire, ST5 5BG, UK. E-mail: [email protected]. Manuscript received 3 Apr. 2002; revised 20 Jan. 2003; accepted 18 Feb. 2003. Recommended for acceptance by M Shepperd. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number 116257. 0098-5589/03/$17.00 ß 2003 IEEE

2

REQUIREMENTS FOR A SOFTWARE BIDDING MODEL

We started our investigation of the software bidding problem by undertaking a survey of bidding practices and portfolio management practices in software and other industries ([21] and [22]). We included portfolio management in our survey because we believe a portfolio viewpoint is needed to manage residual projects risks [13]. As a result of this investigation, we identified a number of requirements that any useful software bidding model ought to address. These requirements are shown in Table 1, which includes a definition of the three categories of risk our bidding model aims to address. Published by the IEEE Computer Society

KITCHENHAM ET AL.: MODELING SOFTWARE BIDDING RISKS

543

TABLE 1 Requirements for a Software Bidding Model

We did not find an existing model that directly fitted the needs of software industry. Of the two models we found most relevant, one assumed a much simpler production situation than is usual for a software company (i.e., one project at a time in strict sequence), but was still unable to provide a closed form model solution [7]; the other proposed a rule-based expert-opinion model, but had not completed model construction [5]. We, therefore, decided to develop a software specific model using the relevant bidding factors from the construction industry.

We identified five possible modeling approaches (see Table 2), four of which had been used in the software industry and one that we found after reviewing oil industry literature [18]. Given our requirements, we opted for using Koller’s modeling approach. Koller’s method also has the advantages both of using a very simple modeling method and of having a well-defined mechanism for modeling Category 3 risks. Koller identifies all the steps required to build a model for a specified purpose [18]. The starting point to developing such a model is by organizing meetings with the

544


VOL. 29,

NO. 6,

JUNE 2003

TABLE 2 Modeling Methods

interested parties in a particular organization and deciding the model’s goals and outputs in collaboration with those parties. Since the origin of our modeling ideas was related literature, not the requirements of a specific organization, we did not perform the requirements gathering and consensus gaining steps, but, thereafter, we followed the process laid out in his book. Thus, our model is not based on a specific situation and it would need to be specialized before use. The initial steps of Koller’s method suggest how such specialization can be performed. Koller uses Monte Carlo simulation to demonstrate the impact of uncertain input values on the model output, which is shown as a frequency distribution or an inverse cumulative function. Koller explicitly avoids using a named probability distribution or specifying a distribution in terms of its formal parameter values, on the grounds that such concepts are not understood by managers. Instead, he recommends defining a skewed distribution in terms of three values: minimum, maximum, and most likely value; a symmetric in terms of two values: minimum and maximum; and a spiked distribution (i.e., where there is no input value uncertainty) in terms of a single value. Fig. 2 illustrates a skewed cost distribution that could be constructed from three values. Koller recommends using a measure of peakedness (assessed on a scale of 1 to 10) to vary the shape of the distribution according to how sure the cost estimators are that the real value will be close to the most likely value [18]. Koller did not define any specific method for determining the distribution from input values, or the peakedness adjustment, nor did he specify any particular simulation package. He recommends constructing the input distributions by Monte Carlo simulation, showing them to the

organization experts, and making any necessary adjustment to the distributions in an ad hoc fashion based on their comments. Our approach has been to: Use the simulation facilities available in the STATATM statistical package because we were already familiar with it. (STATATM is a registered trademark of the STATA Corporation.) . Base skewed distributions on a truncated Gamma distribution by obtaining samples from a standard Gamma distribution, scaling them to the range of input values and rejecting values outside permitted range. . Base symmetrical distributions on a standard Normal (Gaussian) distribution, scaling them to fit the range of values and rejecting any values outside the permitted range. . Use an algorithm for peakedness adjustments that moves a percentage of values closer to the central value for large values of peakedness and moves a percentage of values away from the central value for low peakedness values. The algorithms we used are documented in a Keele Technical report and freely available on the Web [23]. .

3

A GENERIC SOFTWARE BIDDING MODEL

Our model is shown in Fig. 1 as a contributing-factor model. The model has one final output (probability of a successful bid) and several intermediary outputs: supplier price to produce distribution in monetary units, supplier delivery date distribution, client price distribution, and client delivery date distribution. Because it is a generic model,


545

Fig. 1. Generic software bidding model. Factors included in the model are named, arrows show the direction of influence among the factors.

we have suggested example values for various parameters to illustrate how the model works. However, the values would need to be changed if the model were specialized to the context of a particular bidding situation.

3.1 Price Price is the central element of the model. Our basic assumption is that price can be represented as a simple additive equation: Price (£) = Estimated cost (£) + Profit (£) + Contingency (£) (1) There is nothing particularly original about (1). Companies usually include a contingency element in their price. However, in our experience, contingency is not well defined and is estimated by intuition. In addition, we have observed in industry that costs are often estimated in terms of effort that is converted to a price based on “burdened man hours.” Burdened man hour rates include a predetermined profit level preventing bid managers from adjusting profit to match the needs of a specific bidding situation.

3.1.1 Estimated Costs Estimated costs include staff costs and hardware/software costs. These costs are derived from the outline project plan or work breakdown structure. We assume that these costs are represented as an estimate triple (minimum, most likely, and maximum) because software development tasks cannot be costed accurately, particularly during the bidding process when the project is still poorly defined. The distribution represents the uncertainty in the estimates for planned tasks and thus quantifies Category 1 risk. There are many cost estimation models and methods that produce effort estimates in terms of a distribution, or an estimate triple (see, for example, [8], [11], [26]). However, to

be compatible with our bidding model, cost models must have input values that are available during the bidding period. So, for example, models with input values based on measures obtained from detailed requirements specification are not particularly useful. Furthermore, the models must be calibrated to effort values based on planned effort, i.e., effort required to respond to unplanned rare events must be excluded. Thus, in practice, it may be necessary to use expert opinion to obtain the cost estimate triple. Cost models may, however, help identify cost factors (such as staff ability and team effectiveness) that are unknown during bidding and increase the uncertainty in cost estimates. Once the supplier organization determines the risk level for a project, a cost value can be obtained using the estimate triple to simulate the full cost distribution and reading the appropriate value from the cumulative probability distribution. For example, as shown in Fig. 2, a risk level of 70 percent indicates that the organization was prepared to accept a 30 percent risk that the project would cost more than the cost value used to construct the price. Software development staff or managers must determine the cost estimation distribution (i.e., the actual minimum, maximum, and most likely values from which the distribution is constructed). Senior managers must define risk level, in accordance with company policy.

3.1.2 Profit The profit element in the pricing equation is based on the return an organization expects on its projects. We model profit as a percentage of the cost. If profit is accounted for separately from staff rates, it can be adjusted according to: .

The importance of the project to the organization in terms of its strategic business goals. If a project is

546


VOL. 29,

NO. 6,

JUNE 2003

Fig. 2. Risk Level (RL) for a cost distribution.

important, an organization may reduce its required profit level in order to increase the probability that the bid is successful. . The impact of competitors in the bidding process. The presence of competitors may require that the profit level be reduced in order to increase the probability that the bid is successful. We model importance and competition using subjective ordinal scales for each factor and associate an adjustment level with each scale point. We assume that the importance of the project to the organization is assessed on a four point ordinal scale: critical, significant, nominal, and unimportant. Then, for each point on the scale, we define the adjustment that will be made to the profit level. Thus, the equation used to calculate profit is: Profit = Estimated Cost (Profit Level + Importance Adjustment + Competition Adjustment)/100,

(2)

where Profit Level, Competition Adjustment, and Importance Adjustment are percentages and Profit and Estimated costs are monetary values. We use an additive adjustment (as opposed to a multiplicative adjustment) because there may be occasions when a negative profit level is necessary. For a particular project, we might not be sure of its strategic importance (for example, senior management may be undertaking a strategic review, or the project may fit some parts of the company strategy but not others); thus, in our model, we associate a probability with each point on the scale, for example, a user might specify: critical 0.1, significant 0.4, nominal 0.4, unimportant 0.1. In a Monte Carlo simulation, for each iteration, we obtain a random number from a flat (uniform) distribution and use that value to determine the importance value as follows:

If the random variable is in the range < 0.1, importance is assumed to be critical and the adjustment factor is -7 percent. . If the random variable is 0.1 and < 0.5, importance is assumed to be significant and the adjustment factor is -3 percent. . If the random variable is 0.5 and < 0.9, importance is assumed to be nominal and the adjustment factor is 0 percent. . If the random variable is in the range 0.9, importance is assumed to be unimportant and the adjustment factor is +5 percent. The adjustment for competition is calculated in a similar way to the importance adjustment, although we restrict ourselves to three ordinal scale levels: Extensive, Nominal, and Low, with adjustments of -5, 0, and +7 percent, respectively. In a specialized model, the number of scale points and the value of the adjustment factors would be decided by internal company experts. In addition, other adjustment factors might be included such as the type of client, or the state of the economy. .

3.1.3 Contingency Contingency is the cost allocated to handling rare events, i.e., events that are not included in the project plan or work breakdown structure because they are unlikely to occur, for example, office fires, loss of critical staff, and subcontractor business collapse. Thus, contingency is related to Category 2 risks. Like cost estimates, contingency can be represented as a random variable from a distribution. Unlike development costs, contingency costs may not be needed since there is a nonzero probability that the scenario corresponding to the contingency does not occur. In order to use this modeling approach, it is necessary to determine the cost triple for the contingency distribution and the probability of occurrence. Such assessments will


need to be based (at least partially) on expert judgement. Historic data can give some indication of the extent of contingency costs on other projects and the frequency of specific contingency events, but past history would need to be reassessed in the context of a specific project. Using a contingency cost triple in a similar way to the cost triple, the corresponding skewed distribution can be generated using Monte Carlo simulation. To determine the contingency value for the price equation, we use Kitchenham and Linkman’s suggestion that contingency should be regarded as analogous to an insurance premium [13]. We select a random value from the contingency distribution (C) and multiply C by p (the probability that the event relating to the contingency occurs). The resulting value Cp can be regarded as the insurance premium for the project that is charged to the customer. As pointed out by Kitchenham and Linkman, if the contingency actually occurred, it would cost C, not Cp, to recover from. However, if the full cost were included in the price, it would be prohibitively large (and unlikely to win the bid). Furthermore, it would be unreasonable to charge customers for the cost of events that may never happen. Kitchenham and Linkman suggest that the contingency from all projects managed by a particular organization should be held centrally to provide a large enough pool of resources to cater for the proportion of contingencies that actually occur. This implies the contingency element of price (like the profit element) is included in the price, but not in the project budget. The initial contingency, Cp, can be adjusted according to the current status of the organization’s contingency fund. If the contingency fund is low, the contingency for the current project should be increased and vice versa. The adjustment can be calculated using a subjective ordinal-scale assessment of the status of the contingency fund. Note, as we describe later, our validation exercise revealed a flaw in this method of calculating contingency. The correct method of calculating contingency is documented in [21] and [24]. For simplicity, we assume that we have a single contingency distribution associated with a probability of occurrence. However, in practice, contingency assessment may be much more complicated: .

.

.

A company may identify a number of different Category 2 risks each with their own probability of occurrence. In this case, the contingency (i.e., expected loss) for each risk must be aggregated, allowing for any dependencies among individual risks. Some Category 2 risks are duration related rather than cost related. In such cases, the duration slippages must be converted into a cost based on the penalty structure defined by the client. Even if a Category 2 risk is cost related, if the cost is a result of additional staff effort, this may also lead to a delivery delay and an additional penalty. We discuss how this may be modeled in [14].

3.2 Estimated Delivery Date The further out the delivery date quoted in the bid, the less likely it is that a proposal will be accepted by a client; so, like the price, the proposed delivery date impacts the

547

probability that the bid is successful. Since delivery date is assessed by reference to planned work, delivery date uncertainty is related to Category 1 risk. The expected delivery date of a project depends on the amount of work to be done and the team size and the amount of work already being undertaken by the organization. However, this relationship is complex. Many cost estimation methods use an estimate of effort as the basis of an estimate of duration (and, hence, delivery date). In our bidding model, we do not include any relationship between duration and costs. This is not because we do not believe such a relationship exists; it is in order to present our model in its simplest form. In [14], we describe a more detailed model in which the relationship between effort and cost is explicitly modeled. We assume that duration can be modeled as a skewed distribution with parameters (minimum, most likely, maximum, and peakedness) specified by the software production group. We assume that the duration distribution is originally based on the work required to complete the proposed project in the absence of an assessment of total organization workload. This implies that the duration estimate should be adjusted to allow for the organization’s current and expected workload, i.e., if the organization is overstretched, the duration estimate should be increased (for example, Powell describes the effect of competition for resources across a set of projects in a portfolio [25]). We use an ordinal scale measure of total workload and adjust duration in a similar way to the importance assessment for profit level.

3.3 Probability of a Successful Bid Other researchers ([5] and [7]) suggest that probability of success can be assessed from past performance, but we believe the proportion of bids won in the past is not a good estimator of the probability of success of a specific bid. Assuming both that a bid meets the client’s functional and performance requirements and that the supplier meets basic client requirements (e.g., by being on the client’s approved supplier list, or by receiving the invitation to tender in the first place), the chance of a specific bid succeeding should be a function of the extent to which price and delivery date match the client’s price expectations and delivery date requirements. In order to evaluate the probability of success, we suggest representatives of the marketing or sales group must define the “price to win” and “delivery date/duration to win” distributions from their knowledge of the client requirements [3]. The distributions must be specified independently of the price and delivery date estimates estimated by the production group. The price and delivery date to win distributions can be compared with the production group’s evaluation of the price and delivery date needed to complete the required product. Comparing the “price to win” distribution with the “price to produce” distribution, we can calculate the probability that the price to produce will be accepted by the client. This probability is equated to the probability that the bid will be successful relative to price. We can make a

548


VOL. 29,

NO. 6,

JUNE 2003

TABLE 3 Model Evaluation Framework

similar assessment of the probability that the delivery date will be accepted. The overall probability of a successful bid given the price and delivery date needed to produce the required application can be calculated by multiplying the probability of the price being successful by the probability of the delivery date being successful (assuming the probabilities are independent). The probability of success of the bid is related to a Category 3 risk, i.e., the risk that the bidding process is unsuccessful. This method of determining probability of success has not been suggested previously, so we have no concrete evidence that it is viable. It depends critically on the ability of marketing/sales to assess the client’s expectation of price and delivery date. There is evidence that such estimates are made, for example, Boehm [3] includes “Price to win” as an estimating method and DeMarco [6] mentions “political” estimates that are aimed to tell the client or senior manager what they want to hear. We believe such estimates are likely to be a poor basis for deciding on a price, but a good one for determining the probability of success. Furthermore, we have spoken to many estimators who complain that their estimates are arbitrarily overridden by senior managers or marketing staff. Our proposal suggests a mechanism for using marketing estimates and production-based estimates constructively to assess pricing risks and opportunities.

4

MODEL VALIDATION

Our initial validation of the model considered the extent to which it meets its requirements [14]. However, this is a very weak form of validation, so we developed an evaluation framework ([15] and [16]), which is an extension of Lindland et al.’s framework for evaluating the quality of

conceptual models [19]. We incorporated some of Boehm’s criteria for software cost model evaluation [3] (appropriately adapted for price models). We also considered the implications of the evaluation Abdel-Hamid and Malik performed of their systems dynamics model of software development [1] and the evaluation Berry and Vanderbroek performed of a metrics program [2]. The evaluation framework is shown in Table 3. In accordance with Lindland et al.’s advice, we separate the properties of the model that support the quality goal from the actions (means) used to achieve the goal. The quality goals identified in the framework use the concept of feasibility. This means that goals need not be perfectly achieved as long as they reach a state where further work to achieve them is uneconomic. The evaluation framework identifies five quality dimensions for model evaluation. Syntactic Quality evaluation aims to check that all the statements in the model are syntactically correct. Semantic Quality evaluation aims to check that the model is feasibly complete and valid. Completeness means that the model contains all statements about the domain that are relevant to the model. Validity means that all the statements made by the model are correct. Pragmatic Quality evaluation aims to check that the solution is both understandable and understood adequately by its target audience. The quality requirement is stated in terms of feasible understandability and feasible comprehension. Test Quality evaluation aims to check that the model has been adequately tested in terms of feasible test coverage. Value evaluation aims to check the extent to which the use of the model improves the bidding process of the user organization. Fig. 3 shows an evaluation process linked to the model construction and deployment process. This is a simplified model so iterations between steps are not shown.


549

Fig. 3. Model evaluation construction development and evaluation process. Lines with single arrows represent sequential links between process steps, dashed lines with single arrows represent backwards links between processes, lines with double arrows are used to identify linked processes, and lines with a lozenge link a process to its owner.

One very important concept is the concept of a generic versus a specialized model. Although the initial steps in the evaluation process appear the same for generic and specialized models, this conceals one major difference. For our generic bidding model, the domain was the literature on bidding in software and other industries and our personal experience of teaching software practitioners cost estimation and risk management. For a specialized model, the domain would be the specific bidding process of a particular organization plus the generic model, which can be regarded as a partial template for a specialized model. Furthermore, the different nature of the audience for generic and specialized models affects the scope of any evaluation: .

Model Value. We cannot evaluate the Practical Utility (i.e., Value) of a generic model because a generic model would not be deployed, only a specialized model would be deployed and used.

Pragmatic Quality Evaluation. Pragmatic Quality evaluation is divided into feasible comprehension and feasible understandability. We did not attempt to evaluate comprehension for the generic model. It was aimed at academics, consultants, and practitioners who might choose to adopt its ideas. This is a very ill-defined population, which would be difficult to sample in a manner appropriate for a surveybased questionnaire or an experiment. Thus, although we presented our results in a series of seminars, we did not canvas the opinion of the audience because asking a self-selected audience to answer questions about a model, where the questions were posed by the developers of the model themselves, would not result in a convincing demonstration of pragmatic quality. Therefore, we have not attempted to evaluate Model Value or the comprehension part of Pragmatic Quality. Even evaluating Pragmatic Quality from the viewpoint of .

550


understandability is difficult. Although we used the recommended means to achieve understandability, it was not clear to what extent we had actually achieved understandability. It was difficult for us, as model builders, to assess how well we had documented the model since any evaluation of the extent to which understandability has been achieved is inherently subjective and our views were likely to be biased.

4.1 Syntactic Quality Evaluation The syntactic quality evaluation was performed as a manual check on the model implementation documentation [23] and the preliminary model description [14]. This documentation consists of descriptions of the price model, the related profit model and the equations, input and output factors, and the flowcharts of all the simulation programs. This exercise detected eight model defects, of which six were related to semantic quality and two were related to pragmatic understandability [24]. The defects were corrected prior to the Test quality evaluation. 4.2 Semantic Quality Evaluation The semantic quality evaluation involved construction of a series of cross-reference tables that identified concepts related to the generic model domain and how they were dealt with in the model (see [21] and [24]). We constructed four tables specifying: Factors we considered for inclusion in the model on the basis of our personal experience, identifying their importance, and supporting literature. . Factors affecting mark-up (i.e., price) in engineering and manufacturing. We mapped each factor definition to a software equivalent; classified it as a cost, a mark-up, a schedule, or a bidding factor; and provided a justification for inclusion/exclusion in the model (for example, all cost factors were excluded). . Factors used in insurance models to assess premiums, how they applied to the software bidding situation, and whether they were included in the model. . Modeling methods that might be used for a bidding model (see Table 2). After constructing the table describing the insurance industry, it was clear that very few concepts were used in our bidding model. As a result of this fact together with results obtained from the Test quality evaluation, we used Koller’s method to develop a model of the calls on a central contingency fund ([21] and [24]). This showed that Kitchenham and Linkman’s proposal for determining the amount of the contingency element of price ignored the probability of multiple project failures. To address this flaw, we developed an improved algorithm for contingency calculation ([21] and [24]). .

4.3 Test Quality The test quality evaluation involved executing a high-level predefined test plan [16]. The testing activity was divided into three main areas. First, we evaluated the sensitivity and

VOL. 29,

NO. 6,

JUNE 2003

stability of the model in the presence of systematic changes to the input values. Second, we assessed the relative importance of the different factors in the model. Finally, we used predefined bidding scenarios to test the fidelity of the model, where fidelity means that the model outputs are consistent with experience. We used a hypothetical set of input values shown in Table 4 and varied one input value at a time. These tests confirmed that, in most cases, the model was internally consistent. Changes in the setting of the adjustment factors cause proportionate change in the model output distributions. The exception was that the adjustments of the contingency costs and, indeed, the contingency costs themselves did not have any significant effect on the output distribution. This result occurred because (as indicated by the semantic validation and confirmed by the fidelity tests) the contingency premium was underestimated. In order to run the fidelity tests, we needed to determine the value, to the supplier organization, of winning a bid. To do this, we developed a profit model using Koller’s modeling method [18]. This was based on the formula: Profit = Price - Contingency – Costs.

(3)

The price determined from the profit model can be used as an input value and costs and contingency values obtained from sampling the relevant distributions without any constraints (e.g., ignoring the risk value in the cost distribution). The profit model allowed for Category 3 risks (i.e., complete failure of the project). The fidelity tests results are summarized in Table 5. Figs. 4 and 5 refer to fidelity scenarios 1 and 2, respectively. Fig. 4 shows the likely profit both for a project priced on cost alone and for the same project priced on cost, profit level, and contingency. Fig. 5 shows the likely profit both for a small and for a large project. The profit distributions relate to three project outcomes: the project proceeds as planned, the project calls on the contingency fund, or the project fails completely.

5

DISCUSSION

Our model relies on the expertise of its users in terms of their ability to specify the distribution of various random variables. Its advantages compared with using an expert to determine each output directly are that: .

.

.

The model makes it clear both that expertise is needed from different organization groups involved in the bidding process and how that expertise can be integrated. Only in a very small organization would a single person have all the necessary expertise. It identifies factors that contribute to outputs. This allows managers to identify factors (such as profit level) that can be traded off against the output values. The model also makes it clear that changing the profit level does not affect basic costs. It resolves the problem of reconciling price to win and price to produce estimates, regarding both as important inputs to the bidding process, but making


551

TABLE 4 Example Data

In all cases, peakedness is assumed to be average.

.

.

it clear that the former price relates to the client view and the latter to the producer view. It explicitly identifies the need to consider the specific context within which a bid is constructed. In particular, it identifies the need to consider the current and future commitments of an organization when making scheduling decisions about a proposed project. In conjunction with the profit model, it makes the risks inherent in bidding far clearer to managers who have to make bid/no bid decisions. In particular, the profit model indicates the monetary loss that will occur if the project is a complete failure or needs to call upon the contingency fund. In addition, analysis of the contingency fund indicates

the extent to which a project poses a threat to the financial stability of a company. The bidding model has one major theoretical weakness. We assume many of the inputs will be based primarily on expert judgement. However, there is considerable evidence that judgement-based estimates are biased. Makridakis et al. identified 12 common sources of bias in judgementbased forecasts (see [20, chapter 10]). Our model is particularly vulnerable to two types of bias: anchoring and underestimating uncertainty. We assume that marketing price to win estimates and production cost to produce estimates can be kept independent. However, anchoring bias would suggest that, if any information leaked from one group to another (for example, from marketing to production), the production estimates would be affected. Jørgensen

552


TABLE 5 Fidelity Test Results

Fig. 4. Profit distribution for Scenario 1.

Fig. 5. Profit distribution for Scenario 2.

VOL. 29,

NO. 6,

JUNE 2003


and Sjøberg observed this result in software estimation experiment [10]. Estimates were strongly affected by early indicators of the required effort, even when estimators were told that the original estimates were irrelevant. In addition, model users are asked to estimate distributions in terms the range of possible values (minimum to maximum). Makridakis et al. present results showing that people considerably underestimate uncertainty in estimates. To address this problem, Jørgensen and Sjøberg [11] developed a method by which estimation intervals can be based on the estimate accuracy of past similar projects. Another area that may be regarded as contentious is our assumption that the Category 1, 2, and 3 risks can be treated as independent. Our justification is that, from the viewpoint of eventual project outcome, the three categories of risk lead to three (conditionally) independent outcomes, i.e., the project may be a failure; if it is not a complete failure, it may still encounter unplanned events and call upon the contingency fund (in this case, it may be regarded as a qualified success); if it is not a complete failure nor calls upon the contingency fund, it may be regarded as a success. It is clear, however, that the root causes of total failure and qualified success are likely to be related. We use this fact in the profit model to obtain an estimate of loss in the case of project failure by sampling the extreme range of the contingency cost distribution [24]. We derived a model evaluation framework to validate our model. We believe it worked well as a validation framework, but not as well as an evaluation framework. It provided a means of scoping our validation exercise, but its major weakness is that it is not clear that using the specified means to achieve quality actually achieves the desired quality. Using the means to achieve quality increases our confidence that we have achieved the quality goals but does not confirm them. To work well as an evaluation framework, we need to specify mechanisms to confirm achievement of quality goals (as we have done for Value). Furthermore, as model builders, we cannot claim to have a totally objective view of our own work, so the evaluation framework needs to be extended to include a separate Model Evaluator role where the evaluators are independent of the model developers. A serious practical weakness of the model is that only the generic model has been evaluated. We believe it is critical that the model be tested in a real industrial setting to establish whether it has any genuine value to practitioners. Since it is difficult for researchers not to influence industrial case studies if they have a vested interest in the results [17], we hope to undertake evaluation of our model in collaboration with other independent researchers.

553

delivery date, and probability of success, and is intended to assist senior managers in the bid/no-bid decision. The model is a generic model and, as such, is not directly applicable to a specific bidding situation without proper specialization. However, the generic model demonstrates the basic structure of a bidding model and indicates how a specialized model could be built. Furthermore, we have constructed and used an evaluation framework and process to validate our generic model that can also be used to validate any specialized model.

ACKNOWLEDGMENTS This research was funded by the UK Engineering and Physical Science Research Council as the project “Managing Risks across a Portfolio of Projects” GR/M33709/01. The authors would like to thank Professor Robin Bladen-Hovell of the Economics Department at Keele University for reviewing their research concepts.

REFERENCES [1] [2] [3] [4] [5]

[6] [7] [8] [9] [10] [11] [12] [13] [14]

[15]

6

CONCLUSIONS

We have developed a simplified software bidding model that is represented as a contributing-factor diagram. The model is animated by the Monte Carlo simulation that allows the user to visualize the implications of uncertainty in the model inputs and the extent of uncertainty in the model outputs. The model determines an appropriate price,

[16]

[17]

T.K. Abdel-Hamid and S.E. Madnick, “Lessons Learned from Modeling the Dynamics of Software Development,” Comm. ACM, vol. 32, no. 12, pp. 1426-1438, Dec. 1989. M. Berry and M.F. Vanderbroek, “A Targeted Assessment of the Software Measurement Process,” Proc. IEEE Seventh Int’l Software Metrics Symp., pp. 222-235, 2001. B.W. Boehm, Software Engineering Economics. Prentice Hall, 1981. B. Carter, T. Hancock, J-M. Morin, and N. Robins, Introducing RISKMAN Methodology: The European Project Risk Management Methodology. NCC Blackwell, 1994. N.N. Dawood, “A Strategy for Knowledge Elicitation for Developing Integrated Bidding/Production Management Expert System for the Pre-Cast Industry,” Advances in Eng. Software, vol. 25, pp. 225-234, 1996. T. DeMarco, Controlling Software Projects. Prentice-Hall, 1982. F.F. Easton and D.R. Moodie, “Pricing and Lead-Time Decisions for Make-to-Order Forms with Contingent Orders,” European J. Operational Research, vol. 116, no. 2, pp. 305-318, 1999. R. Fairley, “Risk Management for Software Projects,” IEEE Software, vol. 11, no. 3, pp. 57-67, 1994. N.E. Fenton and M. Neil, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp. 675-689, May 1999. M. Jørgensen and D.I.K. Sjøberg, “Impact of Effort Estimates on Software Project Work,” Information and Software Technology, vol. 43, no. 15, pp. 939-948, 2001. M. Jørgensen and D.I.K. Sjøberg, “A Simple Effort Prediction Interval Approach,” Achieving Quality in Information Systems (AquIS), 2002. K. Ka˚nsa˚la, “Integrating Risk Assessment with Cost Estimation,” IEEE Software, vol. 14, no. 3, pp. 61-67, 1997. B. Kitchenham and S.G. Linkman, “Estimates, Uncertainty and Risk,” IEEE Software, vol. 14, no. 3, pp. 69-74, 1997. B. Kitchenham, L. Pickard, S.G. Linkman, and P. Jones, “A Preliminary Risk-Based Software Bidding Model,” Keele University Technical Report TR/SE-0103, Sept. 2001, www.keele.ac. uk/depts/cs/se/e&m/tr0103.pdf. B. Kitchenham, L. Pickard, S.G. Linkman, and P. Jones, “A Framework for Evaluating a Software Bidding Model,” Proc. Empirical Assessment and Evaluation in Software Eng. Conf., Apr. 2002. B. Kitchenham, L. Pickard, S.G. Linkman, and P. Jones, “A Process for Evaluating a Software Bidding Model,” Technical Report TR/ SE-0201, Keele Univ., 2002, www.keele.ac.uk/depts/cs/se/e&m /tr0201.pdf. B. Kitchenham, S.L. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary Guidelines for Empirical Research in Software Engineering,” IEEE Trans. Software Eng., vol. 28, no. 8, pp. 721-734, Aug. 2002.

554


[18] G. Koller, Risk Modelling for Determining Value and Decision Making. Chapman and Hall, 2000. [19] O.I. Lindland, G. Sindre, and A. Solvberg, “Understanding Quality in Conceptual Modeling,” IEEE Software, vol. 11, no. 2, pp. 42-49, 1994. [20] S. Makridakis, S.C. Wheelwright, and R.J. Hyndman, Forecasting Methods and Applications, third ed. John Wiley and Sons, 1998. [21] L. Pickard, B. Kitchenham, S.G. Linkman, and P. Jones, “Can Software Bidding Practices be Improved?—Bidding and Portfolio Management Practices in Software and Other Industries,” ACM Surveys, to be published. [22] L. Pickard, B. Kitchenham, S.G. Linkman, and P. Jones, “Can Software Bidding Practices be Improved?—Bidding and Portfolio Management Practices in Software and Other Industries,” Technical Report TR/SE-0003, Keele Univ., 2000, www.keele.ac. uk/depts/cs/se/e&m/tr0003.pdf. [23] L. Pickard, B. Kitchenham, S.G. Linkman, and P. Jones, “Software Bidding Model Implementation,” Technical Report TR/SE-0202, Keele Univ., 2002, www.keele.ac.uk/depts/cs/se/e&m/ tr0202.pdf. [24] L. Pickard, B. Kitchenham, S.G. Linkman, and P. Jones, “Evaluation of the Software Bidding Model,” Technical Report TR/SE 0204, Keele Univ., 2002, www.keele.ac.uk/depts/cs/se/e&m/ tr0204.pdf. [25] A.L. Powell, “Right on Time: Measuring, Modeling and Managing Time-Constrained Software Development,” PhD Thesis, Dept. of Computer Science, Univ. of York, 2002. [26] I. Stamelos and L. Angelis, “Managing Uncertainty in Project Portfolio Cost Estimation,” Information and Software Technology, vol. 43, pp. 759-768, 2001.

VOL. 29,

NO. 6,

JUNE 2003

Barbara Kitchenham received the PhD degree from the University of Leeds. She is a professor of quantitative software engineering at Keele University. Her main research interest is software metrics and its application to project management, quality control, risk management, and evaluation of software technologies. She is particularly interested in the limitations of technology and the practical problems associated with applying measurement technologies and experimental methods to software engineering. She is a chartered mathematician and fellow of the Institute of Mathematics and Its Applications. She is also a fellow of the Royal Statistical Society. She is a visiting professor at both the University of Bournemouth and the University of Ulster. She is a member of the IEEE Computer Society. Lesley M. Pickard received the PhD degree from City University, London, on the use of software metrics to support project management. She is a fellow of the Royal Statistical Society. She is a research fellow at Keele University. Her main research interest is in software measurements and their use in supporting project and risk management. She is also interested in the development of appropriate data analysis protocols for software data sets. Stephen Linkman is a principal researcher of software engineering in the Computer Science Department at the University of Keele. He is a graduate of Leicester University and has worked in both industry and academia for the last 30 years publishing a number of papers as a result of this work. His research interests are metrics and measurement, evaluation in all its forms, and the control and management of software projects.

Peter W. Jones received the PhD degree in 1974 from the University of Wales, Aberystwyth. He is currently a professor of statistics in the Mathematics Department at Keele University. Previously, he was a senior lecturer and reader in mathematical statistics at Keele University and a lecturer in statistics at the University of Wales, Aberystwyth. He is also a nonexecutive director on the board of the Orthopaedic Hospital in Oswestry, UK. Peter is an associate editor of the journal Biometrics, a member of the advisory board of the Egyptian Journal of Mathematics and Statistics, and an editor of the Journal of Apicultural Research. His research interests cover a wide range from theoretical developments in Bayesian sequential estimation and optimization, especially the study of bandit processes, to applications in medicine. In recent years, his work has concentrated on modeling genetic effects of susceptibility and severity of disease, meta analysis, the planning of clinical trials, and the development of clinical scoring models for disease progression. He is an author or coauthor of more than 200 papers.

. For more information on this or any computing topic, please visit our Digital Library at http://computer.org/publications/dlib.