Comparing Some Neural Network Models for Software Development Effort Prediction Mrinal Kanti Ghose, Roheet Bhatnagar
Vandana Bhattacharjee
Department of Computer Science & Engineering Sikkim Manipal Institute of Technology, Majitar, Rangpo East Sikkim, India
[email protected],
[email protected]
Department of Computer Science & Engineering Bitec, Lalpur, BIT Mesra Ranchi, India
[email protected]
Abstract—Software development effort estimation is one of the most major activities in software project management. A number of models have been proposed to make software effort estimations but still no single model can predict the effort accurately. The need for accurate effort estimation in software industry is still a challenge. Artificial Neural Network models can be used for carrying out the effort estimations for developing a software project & this field of Soft Computing is suitable in effort estimations. The present paper is concerned with comparing the results of various artificial neural network models for predicting the software development effort estimation. The neural network models available in MATLAB neural network tools were used and the standard dataset as compiled by Lorenz et.al. was used in the present study. The results were analyzed using four different criterions MRE, MMRE, BRE and Pred. It is observed that the Generalised Regression Neural Network model provided better results. Keywords-Effort Estimation, Artificial Neural Network, Soft Computing
I.
INTRODUCTION
One of the most critical and extremely crucial activity of any software development project is software effort & cost estimation. Accurate estimation of a software development effort is critical for good management decision making. The development costs tend to increase as the complexities in software project development increases. Therefore an important objective of the software engineering community has been to develop an appropriate model to estimate the software effort accurately. But so far after so much of research in this area none of the proposed methods can accurately predict the development costs. Software community has developed unique tools and techniques such as size, effort, and cost estimation techniques and tools to address challenges facing the management of software development projects [1][2][3]. These tools and techniques are utilized for software development phases starting with the software requirements specification. As demand for software applications increases
978-1-4244-9581-8/11/$26.00 © 2011 IEEE
continually and the software scope & complexity become higher than ever, the software companies are in real need of accurate estimates of the project under development. Indeed, good software effort estimates are critical to both companies and clients. They can be used for making request for proposals, contract negotiations, planning, monitoring and control [4]. Molokken and Jorgensen report in their review of estimation studies that software projects expand on average 30-40% more effort than is estimated [5]. II.
METHODOLOGY
In the present work of our research we have tried to find out the Development Time (DT’) by applying the Feed Forward Back Propagation Neural Network model, Cascaded Feed Forward Back Propagation Neural Network model, Elman Back Propagation Neural Network model, Layer Recurrent Neural Network model & Generalised Regression Neural Network model present in the Neural Network toolbox of Matlab 7.5. The standard dataset as proposed by Lopez-Martin et.al. has been used for the experimentation purposes. They used the sets of system development projects, where the Development Time (DT), Dhama Coupling (DC), McCabe Complexity (MC) and the Lines of Code (LOC) metrices were registered for 41 modules. Since all the programs were written in Pascal, the module categories mostly belong to procedures and functions. The development time of each of the forty-one modules were registered including five phases: requirements understanding, algorithm design, coding, compiling and testing [6, 7]. Table I shows the dataset used for carrying out experimentation. All these models were trained with first 31 inputs from the standard dataset and later 10 inputs from the same dataset were used to test the models. The effectiveness of neural network model over the regression analysis model has already been proved with the same dataset[8]. The Development Time (DT’) were obtained for each of the trained neural network models and finally a comparative analysis was carried out based on the standard assessment criterions like MRE, MMRE, BRE and Pred(25).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
Module Description Calculates t value Inserts a new element in a linked list Calculates a value according to normal distribution equation Calculates the variance Generates range square root Determines both minimum and maximum values from a stored linked list Turns each linked list value into its z value Copies a list of values from a file to an array Determines parity of a number Defines segment limits From two lists (X and Y), returns the product of all xi and yi values Calculates a sum from a vector and its average Calculates q values Generates the sum of a vector components Calculates the sum of a vector values square Calculates the average of the linked list values Counts the number of lines of code including blanks and comments Prints values non zero of a linked list Stores values into a matrix Generates range square root Returns the number of elements in a linked list Calculates the sum of odd segments (Simpson’s formula) Calculates the sum of pair segments (Simpson’s formula) Generates the standard deviation of the linked list values Returns the sum of square roots of a list values Prints a matrix Calculates the sum of odd segments (Simpson’s formula) Calculates the sum of pair segments (Simpson’s formula) Calculates the average of linked list values Returns the sum of a list of values Generates the standard deviation of linked list values Prints a linked list Calculates gamma value (G) Calculates the average of vector components Calculates the range standard deviation Calculates beta 1 value Returns the product between values of two vectors and the number of these pairs Counts commented lines Reduces final matrix (according to Gauss method) Reduces a matrix (according to Gauss method) Counts blank lines
MC 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 5 5 5
DC 0.25 0.25 0.333 0.083 0.111 0.125 0.125 0.125 0.167 0.167 0.167 0.167 0.167 0.2 0.2 0.2 0.2 0.25 0.25 0.083 0.125 0.125 0.125 0.143 0.143 0.143 0.143 0.143 0.167 0.167 0.2 0.25 0.25 0.25 0.077 0.077 0.111 0.2 0.143 0.143 0.2
TABLE I : STANDARD DATASET DEVELOPED BY LOPEZ ET. AL. MC: MCCABE COMPLEXITY, DC: DHAMA COUPLING, LOC: LINES OF CODE, DT: DEVELOPMENT TIME (MINUTES)
LOC 4 10 4 10 23 9 9 14 7 8 10 10 10 10 10 10 15 10 10 17 11 15 15 13 14 14 15 15 13 14 18 9 12 17 16 31 16 24 22 22 22
DT in minutes 13 13 9 15 15 15 16 16 16 18 15 15 18 13 14 15 13 12 12 22 19 18 19 21 20 21 19 20 15 13 19 13 12 12 21 21 19 18 24 25 18
III.
VARIOUS CRITERIONS FOR ASSESSMENT OF ESTIMATION MODEL
There are many evaluation criteria to evaluate the accuracy of the software development effort in literature. The Mean Magnitude Relative error (MMRE) is a widely-accepted criterion in the literature and is based on the calculation of the magnitude relative error (MRE). Eq. (1) as below shows an equation for computing the MRE value that is used to assess the accuracies of the effort estimates. Here, the Yj represents the actual effort and j.
is the estimated effort for the project
is more likely to select the true model if the under-estimate is served. Either the MMRE or the MdMRE aggregates the multiple observations. In this study, the MdMRE, MMRE, BRE and Pred(0.25) were adopted as the indicators of the accuracy of the established software effort estimation models since they are the ones most widely used in the literature, thereby rendering our results more comparable to those of other work. IV.
EXPERIMENTAL RESULTS
Table II summarises the development time estimates as obtained for different neural networks. TABLE II: ESTIMATED DEVELOPMENT TIME (DT’) OF VARIOUS MODELS
Eq. (1) The MRE calculates each project in a dataset while the MMRE aggregates the multiple projects. The model with the lowest MMRE is considered the best [9]. As shown in Eq. (2), the estimation accuracy of the MMRE is the mean of all the MREs among n software projects. For example, an MMRE with a value of 0.5 means that the esti-mate matches 50% of the actual effort on average.
Eq. (2) Another measure of Pred(l) was also adopted to evaluate the performance of the established software effort esti-mation models. This measure is also often used in the literature and is a proportion of a given level l in the accuracy. The equation for computing Pred(l) is shown in Eq. (3), where n is the total number of observations and k is the number of observations with an MRE less than or equal to l. An acceptable value for l is 0.25 in the literature, which is also adopted in this study.
31
19
9.04
9.54
9
9.49
DT’ Using Genera lised Regres sion NN 21.93
32
13
9
9.84
9
9
14.98
33
12
9
21.91
9.12
9
17.68
34
12
9
9
9
9
21.16
35
21
21.98
22
20.05
9.21
19.33
36
21
21.99
22
22
18.89
15
37
19
21.97
21.87
20.16
9.17
19.33
38
18
21.99
9.71
9
19.19
15
39
24
9.31
9
9
9.12
15
40
25
9.31
9
9
9.12
15
41
18
9
9
9
9
15
Serial Num ber
Actual DT
DT’ Using FFBP NN
DT’ Using Cascad ed FFBP NN
DT’ Using Elman BPNN
DT’ Using Layer Recurr ent NN
Following Table III gives the comparison of various neural network models based on the standard evaluation criterions. TABLE III: COMAPARISON OF VARIOUS MODELS
Eq. (3) Balance Relative Error (BRE) is yet another evaluation criteria used for estimation accuracy and is defined as given in Eq (4) below
BRE (%) = 100 *
E Eˆ min(E , Eˆ )
ˆ where E = estimated effort and E = actual effort Another often-used evaluation criterion is the median MRE (MdMRE). The MMRE is fairly conservative, with a bias against overestimates, while Pred(0.25) identifies the prediction systems that are generally accurate. The MdMRE is less sensitive to the extreme values compared to the MMRE [10]. It exhibits a similar pattern to MMRE but it than the other Neural Network models as it scores over the other models on all the four evaluation criterion.
Neural Network Model FFBPNN Cascaded FFBPNN Elman BPNN Layer Recurrent NN Generalised Regression NN
Mean MMRE(%)
MdMRE
Mean BRE(%)
Pred (0.25)(%)
32.21
9.31
0.6316
55
38.99
9.71
0.7287
45
34.03
9
0.7095
45
39.17
9.12
0.8196
45
27.58
15
0.3376
64
A model which gives lower BRE is better than that which gives higher BRE. A model which gives higher Pred(n) is better than that which gives lower Pred (n). A model which gives lower MMRE is better than that which gives higher MMRE. Hence from the Table III we can observe that Generalised Regression Neural Network (GRNN) is better Fig. 1 below shows the comparison of different neural network models based on MMRE, MdMRE and Pred(0.25) and we can
see that the Pred(0.25) and MdMRE values in case of GRNN model is highest among all the models, so it can be concluded that GRNN model is the better model for effort estimation among all other models.
accurate effort estimate. Homogeneous dataset results in better & more accurate effort estimates while the irrelevant & chaotic dataset results in lesser accuracy in effort estimations. The aim of this study is to explore the effect of the estimation accuracies of different neural network models in estimating the development time from the standard dataset developed by Lopez et. al. Four widely-used evaluation indicators, MMRE, MdMRE, BRE and Pred(0.25), were adopted in the present work. These indicators show that GRNN model is the best model among all the other chosen model. Our future work may apply more evaluation indicators to assess the estimation abilities of different models on different dataset. The same experiments as those used in this study are required to explore the effect of accuracies of the software effort estimation models using different sources of historical software projects as historical data is very much important in carrying out the software development effort estimations. REFERENCES
Fig. 1 Comparison of different models based on MMRE (%), MdMRE And Pred(0.25)
Fig. 2 Comparison of different models based on BRE (%)
Fig. 2 above shows the comparison of different neural network models based on BRE (%) and from the bar graph we can see that GRNN model has the lowest value and hence it is the best model for effort estimation among all the models. V.
CONCLUSION
A crucial factor affecting the success of a software development project is the ability to derive a more reliable and
[ 1] Jones, T.C. Estimating Software Costs, McGraw-Hill, 1998. [ 2] Thayer, H.R., Software Engineering Project Management, Second Edition IEEE CS Press, 2001. [ 3] C. Symons, “Come Back Function Point Analysis (Modernized) – All is Forgiven!”, Proc. of the 4th European Conference on Software Measurement and ICT Control, FESMA-DASMA 2001, pp. 413-426, 2001. [ 4] M. W. Nisar, and Y.-J. Wang, and M. Elahi, "Software Development Effort Estimation Using Fuzzy Logic - A Survey", in 5th International Conference on Fuzzy Systems and Knowledge Discovery, 2008, pp. 421-427. [ 5] K. Molokken, and M. Jorgensen, "A Review of Surveys on Software Effort Estimation", in International Symposium on Empirical Software Engineering, 2003, pp. 223-231. [ 6] C. Lopez-Martin, C.Yanez-Marquez, A.Gutierrez-Tornes, “Predictive accuracy comparison of fuzzy models for software development effort of small programs, The journal of systems and software”, Vol. 81, Issue 6, 2008, pp. 949-960. [ 7] C.L. Martin, J.L. Pasquier, M.C. Yanez, T.A. Gutierrez, “Software Development Effort Estimation Using Fuzzy Logic: A Case Study”, IEEE Proceedings of the Sixth Mexican International Conference on Computer Science (ENC’05), 2005, pp. 113-120. [ 8] Roheet Bhatnagar et. al. "Software Development Effort Estimation – Neural Network Vs. Regression Modeling Approach" in International Journal of Engineering Science and Technology (IJEST) Vol. 2(7), 2010, 2950-2956 [ 9] M. Shepperd, C. Schofield, Estimating software project effort using analogies, IEEE Transactions on Software Engineering 23 (12) (1997)736–743. [ 10] T. Foss, E. Stensrud, B. Kitchenham, I. Myrtveit, A simulation study of the model evaluation criterion MMRE, IEEE Transactions on Software Engineering 29 (11) (2003).
