An Empirical evaluation of Effort Estimation Models in ...

An Empirical evaluation of Effort Estimation Models in Software Engineering Projects Abid Ali Northern university Hakeem Abad, Nowshera, Pakistan [email protected]

Abstract- The nature of software is different from

This estimation depends on the quantity of information available, but in early stages of projects there is very little information. These information lacking may cause the under and over estimation of software projects which leads very costly errors which include projects rejection, omitting of important features and abandoned of projects. Getting these accurate project estimation reduce unnecessary cost and increase efficiency and effectiveness of organization. An expert project manager can do wise decisions about the current projects regarding cost estimation but still uncertainty is there. Project manager is human being and cannot do intensive computations without a tool that can support his decision, and there is no standard method for expert project manager opinion-based prediction nor accurately, he can identify an appropriate estimation for a new project [1]. In 2009, Magne and Stein built up a new model called BEST to assist the project manager in his judgment regarding effort estimation of software projects [2]. The project manager use the software estimation by analogy where comparison of current project with previously completed projects is carried out. Expert opinion based approaches depends on personal experience and analogy based approaches depends in intensive data. In Software Development Effort estimation, we model the work hours prediction and number of developers to develop the project. In project management, the effort invested represents the most important factor and is measured in Person-Months. Effort estimation can be used to predict project cost and schedule. The project planning, iteration planning, pricing and bidding processes and investment analysis can be inputted by these estimates. Project managers continuously trying to make reliable project estimates. Jorgensen [4] analyzed that 60-80% of the software projects will overrun their estimated time and cost by 30%. Some other authors [5] also concluded that there is intensive need of good estimation techniques. There are many Software Effort Estimation models namely Algorithmic, Expert judgment based and Non Algorithmic. As stated earlier, the unique nature of

other engineering artifacts. The parameters like complexity, conformity, non-visibility and changeability makes the software nature more different from other engineering products. The software project managers have many and needs extra ordinary skills in managing software projects. One of these issues is software projects effort estimation. There are number of academic efforts in providing different software project estimation models to help project managers in to tackle this challenging issue. These models have advantages and disadvantages and have different level of performance in different situations. There are multiple and conflictive requirements for selecting estimation model for project at hand. Multi Criterion Decision Making approaches are best tools for handing such situations. This work applies the ANP (Analytic Network Process) for selection and prioritization of cost estimation models. Keywords: Effort Estimation, Analytic Network Process, Multiple and Conflictive Requirements

1. Introduction The process of developing software becoming more expensive and it represent an expensive cost factor of any information system. The quality of firm’s decisions about investment in projects depends on the accuracy of estimation. The firm’s software investment decisions are very complex because estimation of software projects is very technical activity and needs very high profile skills and expertise. The complexity, unique nature, invisibility and strong modules interdependency of software product makes the estimation of software projects more unpredictable. This unpredictability inversely affects the quality of decision making during contract biding or committing required resources. The literature shows that 60-70% of software projects are failed and this fact exerts more pressure on software project managers. The top goal of software project manager is to accurately estimate the new project.

1

software makes decision making more complex and this complexity exponentially increases when there are multiple criteria. The class of MCDM mechanisms provide best tool to handle such issues very nicely. This work taking attributes of software effort estimation models and apply ANP (Analytic Network Process) which is one of the MCDM mechanisms, to analyze the algorithmic, expert judgment and non-algorithmic estimation models on the basis of relative intensity of these attributes. This paper can be structured as: the next part represents the literature survey which justifies the problem and solution at hand. Part 3 explains the evaluation framework. Then research methodology and results are analyzed and at last there relevant discussion and conclusion is presented.

of criteria (attributes) which can be used for the evaluation of effort estimation models, the second part is to identify the most practiced models from industry and literature, and the third part is to take one of the MCDM approach to determine the relative values of each attributes and then prioritize the reported models on the basis of these relative values. In order to execute first, a through literature survey was conducted and expert discussions were carried out. For the execution of second part, a literature and industry survey was applied. For the third part, a gap was identified from literature. Previous approaches use fuzzy AHP which provide hierarchical structure to model the problem. In hierarchical structure, the upper layers depend on the lower layers. To model the problem using network structure, where dependency between different clusters and elements of clusters is modeled, we use ANP which one of the MCDM mechanisms. The following attributes are reported from literature: a. Reliability (C1):- observing behavior under different conditions in order to verify design perfection b. Mean Magnitude of Relative Error (MMRE) (C2) :- It can be calculates as [(actual estimate)/actual] c. Pred(x/100) (C3):-: It is percentage of projects for which estimate is within x% of the actual d. d. Uncertainty (C4):- small errors, how the estimates changes

2. Literature Survey Literature report many efforts for improving the quality of decision making by enhancing the level of information in term of cost estimation during software projects. There are many techniques devised for calculating software effort estimation during software projects. Out of these algorithmic, expert judgment and non algorithmic models are well known and are used in industry. These three types of classes have their own benefits and limitations. The expert judgment depends on the project manager experience, personal judgment and soft abilities. In algorithmic approaches, there are some mathematical formulas are used. To derive these models, there is some statistical data analysis used. There are new techniques applied to enhance quality of estimation information. [6] Studied and proposed the application of machine learning approaches for predicting the effort of software projects. [7] Systematically reviewed the empirical evidence to find out whether FWT (feature weighting techniques) leads to superior predictions. In addition it also evaluates these techniques from the perspectives of (i) strengths (ii) approach and weaknesses (iii) performance and (iv) experimental evaluation approach including the data sets used. All these approaches have benefits and limitations according to situation and scenarios. Different estimation follows different ideology. In some situation more than one estimate are applied and in some situations, multiple approaches are combined. So it can be interpreted that no single technique is best for all [8]. Meli and Menzies [9, 10] have suggested such an integrated approach by combining expert judgment with models.

3.2Effort Estimation Models The software development process consists of different activities where each activity has specific goals, inherited complexity and level of importance. Software effort estimation is one of these most important activities. There are different models used having their own advantages and disadvantages. These models can be categorized as algorithmic, expert judgment and non algorithmic. 3.2.1 Expert Judgment In this approach, the efforts of software projects are estimated using expert(s) which have knowledge of application domain and development environment. These estimates based on expert judgment where the process of generating an estimate is not recoverable and explicit, and have strong dependency on expert intuition [11]. According to Jorgensen claim formal models have not replacement strategy but support expert prediction [4]. There is strong willingness in software industry to accept, understand and properly used expert base estimations [12]. During expert judgment, the software engineer or senior project manager provide input [13] but not necessary that

3. Evaluation Framework 3.1Criteria of Estimation Models This study consists of three parts i.e. identification

2

they should be good estimators [14]. In some cases there is under optimism or over optimism is observed during expert judgment process. A study is conducted by [19] in the specific areas in which expert judgment both succeeds and fails, and what commonalities underpinned the disciplines that consistently displayed successful and un-successful expert judgment. A methodology is proposed by [16] to improve the expert judgment. Another methodology which is based upon the findings of Jorgensen [17] and Bolger and Wright [14], which can be divided into three phases: record, learn and estimate. This methodology suggests that with increase in the size of a work item, the application potential and the importance of the estimation increases. According to Jarabek [16], top down strategy is followed for estimation of large work items where large amount of estimate information has been recorded. And bottom up estimation strategy is followed for smaller work items, where less information is obtained.

There are human beings involved in decisions making where the factor of subjectivity exist. To models all these issues, MCDM mechanisms provide best tools. The Analytic Network Process (ANP) is one of these approaches introduced by [25]. This technique model the relation between clusters and nodes of clusters in order to show inner and outer dependency [26]. This technique uses the concept of relativity i.e. to show the relative influence of each cluster over other clusters and each node influence on elements of the same cluster and outer clusters. The nodes are compared with respect to other nodes which is called control criteria. The selection of these control criteria depends on user requirements. The ANP can be characterized by two networking groups. The first group contains the criteria and sub-criteria and second group contains the mutual influence network for the elements which are lying in criteria and sub-criteria groups. This is the reason why the mode of thinking used in ANP is capable of mimicking human thinking more than AHP in decision making [20, 28].

3.2.2 Model Based In these methods, algorithms are used to make estimates about new projects. The past data have much importance and can be used extensively as input to these algorithms. Most of these estimation models depend on size measure i.e. Function Point (FP) and Line of Code (LOC). The size estimation accuracy has direct relation with effort estimation accuracy. These models can be categorized as: algorithmic and non-algorithmic. The most widely used algorithmic model are COCOMO [18], Putnam‟s SLIM [19] and Albrecht‟s Function Point. These algorithmic models have low potential in modeling inherent complex relationship between different contributing factors, to handle categorical data and have lack of reasoning capabilities. On other hands to overtake these limitations, non-algorithmic techniques are used which are based on soft computing [21]. The soft computing techniques can be evolutionary computation, neural networks, and fuzzy logic models and case based reasoning [21].These techniques have well defined applications in other areas. In these model based approaches available historical data can be used in efficient way to effectively evaluate the models [22]. [23] Proposed a model where PSO (Particle Swarm Optimization) is used to adjust simple COCOMO parameters to get better estimates. A data set from NASA projects was taken to evaluate the performance of the model [24].

4

Research Methodology

4.1 Sample Selection In organizations, decision making is executive level activity where different stack holder’s interests are kept in mind. These interests have some time conflicts which make decision making more complex. A proper profiling was done in order to select more experience people and domain expert. The concept and purpose of the study was discussed with people before filling questionnaire. The whole purpose behind this process is to get more consistent data. 4.2 Questionnaire Design In case of MCDM mechanisms, there is a direct comparison between different factors just to measure the influence of one factor over other. There is relativity in these influences i.e. one factor can influence another by some extent but influence the other factor by different extent. The size of questionnaire is a serious problem in case of MCDM mechanisms. There are many comparisons in order to know the relative influence among different elements. To make short the size of questionnaire, mathematical formulas such as transitive property and reciprocal properties are used in designing the questionnaire. For designing questionnaire and specifying questions, the AHP format was used. In this format we compare two factors in term of their relative value using five level likert scale. Figure 3 shows format of the question used for AHP.

3.3 Analytic Network Process The decision in organization is very complex activity and this complexity exponentially increases, when there is conflict in decision parameters.

3

Evaluation of software cost estimation models

Reliability Algorithmic models Mean magnitude of relative error (MMRE)

Expert judgment

Pred (x/100) Non algorithmic models Uncertainty

Fig. 2. ANP Network Process 9

7 5

3 equal 3

5

7

9

4.3 Research Questions This study focusing to answer the following two questions: Q.1 What is relative influence among the criteria used to select software effort estimation model? Q.2 Which effort estimation technique is more appropriate on the basis of the values obtained in Q1?

Extreme preference

Extreme preference

Fig.3. AHP questionnaire format

Table 1 shows satty’s scale which measure relative importance of one factor over other. Table 1. Satty’s scale Intensity Definition level

1

Equal preference

3

Somewhat more preference

5

Much more preference Very much more preference

7

9 2,4,6,8

Absolutely more preference Intermediate values

5. Results The ANP (Analytic Network Process) is a mathematical technique where huge calculation is done to produce results. There are some tools used for these calculations. This study uses ANP solver tool to analyze the input data. Two sources were used for data collection. The experts were used as primary data source to get inner and outer dependency. The literature and case studies were used as secondary data source to validate these results. Table 2 shows the Supermatrix obtained using ANP solver tool. Table 3 and 6 shows the weighted super matrix and limit matrix respectively. These tables show the quantitative values of inner and outer influence among different clusters and clusters elements.

Explanation

Two factors equally preferred. the objective One is slightly favored over other on the basis of judgment and experience one is strongly favored over other Very strongly preference

5.1Prioritization of Criteria The purpose of this study is to show the relative comparisons among different criteria and sub criteria. Table 4 shows this prioritization. 5.2 Prioritization of Estimation Techniques Table 5 shows the ranking of effort estimation models based on the values of relative influences of criteria.

Extreme preference When compromise is done

4

Table 2. Supermatrix Reliability

Mean magnitude of relative error (MMRE)

Pred( x/100)

Uncertainty

Algorith mic models

Expert judgment

Non algorithmic models

Reliability

0

0.174

0.134

0.109

0.146

0.146

0.146

Mean magnitude of relative error (MMRE) Pred(x/100)

0.467

0

0.747

0.476

0.397

0.397

0.397

0.467

0.695

0

0.415

0.397

0.397

0.397

Uncertainty

0.067

0.131

0.119

0

0.06

0.06

0.06

Algorithmic models Expert judgment Non algorithmic models

0.637

0.486

0.528

0.714

0

0.75

0.833

0.104

0.141

0.14

0.143

0.25

0

0.167

0.259

0.372

0.333

0.143

0.75

0.25

0

Table 3.Weighted Supermatrix

Reliability

Reliability

0

Mean magnitude of relative error (MMRE) 0.087

Pred(x/100)

Uncertainty

Algorithmic models

Expert judgment


0.067

0.054

0.073

0.073

0.073


0.233

0

0.374

0.238

0.198

0.198

0.198

0.233

0.348

0

0.208

0.198

0.198

0.198

Uncertainty

0.033

0.066

0.06

0

0.03

0.03

0.03


0.318

0.243

0.264

0.357

0

0.375

0.417

0.052

0.071

0.07

0.071

0.125

0

0.083

0.13

0.186

0.166

0.071

0.375

0.125

0

Table 4. Criteria prioritization

Table 5. Effort estimation models prioritization

Criteria

Ranking

Reliability Mean magnitude of relative error (MMRE) Pred(x/100) Uncertainty

3 1

Relative value 0.069 0.197

2 4

0.192 0.042

5

Estimation model Algorithmic models Expert judgment

Relative value 0.24

ranking

0.079

3


0.181

2

1

Table 6. Limit matrix

Reliability

Reliability

0.069

Mean magnitude of relative error (MMRE) 0.069

Pred(x/ 100)

Uncertain ty

Algorithmic models

Expert judgment

0.069

0.069

0.069

0.069

Non algorithmic models 0.069


0.197

0.197

0.197

0.197

0.197

0.197

0.197

0.192

0.192

0.192

0.192

0.192

0.192

0.192

Uncertainty

0.042

0.042

0.042

0.042

0.042

0.042

0.042


0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.079

0.079

0.079

0.079

0.079

0.079

0.079

0.181

0.181

0.181

0.181

0.181

0.181

0.181

[5]

6. Conclusion and Future Work The results presented in table 5 & 6 can be interpreted as: the expert prefers MMRE and Pred criterion in case selecting software effort estimation model, and algorithmic models are best choices in industry. The limitations of this work are the application of set theory. The concept behind set theory is yes or no. There is lack of some fuzzy approach to narrow the member ship of some property. Similarly by increasing or decreasing the influence of individual criteria, we observe the consequential changes of the priorities and the position of the alternatives. In future fuzziness should be incorporated to this technique to narrow membership different factor, and sensitivity analysis should be done.

[6]

[7]

[8]

References

[9]

Keung, Jacky.: Software Development Cost Estimation Using Analogy: A Review, Software Engineering Conference, 2009. ASWEC'09. Australian. IEEE, 2009 [2] M. Jørgensen and S. Grimstad.: Software Development Effort Estimation: Demystifying and Improving Expert Estimation, In: Simula Research Laboratory - by thinking constantly about it, ed. by AslakTveito, Are Magnus Bruaset, OlavLysne. Springer, Heidelberg, chap. 26, 381-404, 2009. [3] K. Molokken and M. Jorgensen: A review of surveys on software effort estimation, at ISESE 03, 2003. [4] C. Jeganathan.: Developing of fuzzy logic architecture to assess sustainability of the Forest Management, Master thesis, Netherland, 2003. [1]

[10]

[11]

[12]

[13]

6

Juan J. Cuadrado-Gallegoa, b, Pablo RodríguezSoriaa, Borja Martín-Herreraa.: Analogies and differences between Machine Learning and Expert based Software Project Effort Estimation, 2010 11th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, 269-276. Martin Shepperd and Michelle Cartwright. Predicting with sparse data. IEEE Trans. Softw.Eng., 27(11), 987– 998, 2001. Sigweni,b, Shepperd,M.: Feature Weighting Techniques for CBR in Software Effort Estimation Studies: A Review and Empirical Evaluation, PROMISE ‘14, September 17 2014, Torino, Italy, ACM. Roberto Meli. Human factors and analytical models in software estimation: An integration perspective. In Proceedings of the ESCOM-SCOPE 2000, 33–40, Munich, Germany, 2000. Shaker Publishing. T. Menzies, Z. Chen, J. Hihn, and K. Lum.: Selecting Best practices for effort estimation, IEEE Transaction on Software Engineering, Vol 32, No 11. November 2006. C. Jeganathan.: Developing of fuzzy logic architecture to assess sustainability of the Forest Management, Master thesis, Netherland, 2003. Magne Jørgensen and Barry Boehm.: Viewpoints Software Development Effort Estimation: Formal models or Expert Judgment? , Software, IEEE 2009 Volume: 26 Issue: 2, 14 – 19. K. Moløkken and M. Jørgensen.: Expert Estimation of Web-Development Projects: Are Software Professionals in Technical Roles More Optimistic Than Those in Non- Technical Roles? Empirical Software Engineering, vol. 10, 7-29, 2005. Chiou, H.K. and Tzeng, G.H. (2002). : Fuzzy Multicriteria decision making approach to analysis and

[14]

[15]

[16]

[17] [18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

evaluation of green engineering for industry, Environmental Management, 30(6): 816-830. F. Bolger and G. Wright.: Assessing the quality of expert judgment: Issues and Analysis, Decision Support Systems, vol. 11, 1-24, 1994. Jarabek,C: Expert Judgment in Software Effort Estimation, Non-peer Reviewed Writings Department of Computer Science University of Calgary. M. Jørgensen.: A Review of Studies on Expert Estimation of Software Development Effort, the Journal of Systems and Software, vol. 70, 37-60, 2004. Boehm. B. W.: Software engineering economics, Englewood Cliffs, NJ: Prentice-Hall, 1981. Putnam, L. H.: A general empirical solution to the macro software sizing and estimating problem, IEEE Transactions on Software Engineering, SE-4(4) , 345361, 1987. Wang, L., and T. Raz.: Analytic Hierarchy Process Based on Data Flow Problem, Computers & Industrial Engineering, 20, 355-365, 1991. Sehra S.K., Sehra S.S., Kaur R.: A survey of techniques for software project effort estimation, International Conference on Wireless Networks & Embedded Systems (WECON-2009) held on 23-24 October, 2009 at Chitkara Institute of Engineering & Technology, Rajpura, (Punjab), 207-210. Gruschke,T and Jorgensen,M.: Assessing uncertainty of software development effort estimates: The learning from outcome feedback. In METRICS ‟05: Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS‟05), page 4, Washington, DC, USA, 2005, IEEE Computer Society Sheta, A, Rine, D and Ayesh,A.: Development of Software Effort and Schedule Estimation Models Using Soft Computing Techniques, Evolutionary Computation, 2008. CEC 2008. (IEEE World Congress on Computational Intelligence). June 2008 , 1283 – 1289. J. W. Bailey and V. R. Basili.: A meta model for software development resource expenditure, in Proceedings of the International Conference on Software Engineering, 107–115, 1981. T.L. Saaty.: Decision making with dependence and feedback the analytic network process, 2nd edn. RWS, Pittsburgh. Saaty T.L, Vargas T.L.: Decision making with the analytic networking process, Springer’s International Series, New York. MEADE L, SARKIS J (1999): Analyzing organizational project alternatives for agile manufacturing processes: An analytic network approach, International Journal of Production Research, 37(2), 241−261. WANG. L.F (2001): The theory and algorithm of ANP” System Engineering Theories and Practices, 44−50.

7