A neuro-fuzzy model for software cost estimation - IEEE Xplore

A Neuro-Fuzzy Model for Software Cost Estimation Xishi Huang, Luiz F. Capretz, Jing Ren Dept. of Electrical & Computer Engineering University of Western Ontario London, Ontario, N6G 1H1, Canada {xhuang22, lcapretz, jren2}@uwo.ca

Abstract A novel neuro-fuzzy Constructive Cost Model (COCOMO) for software estimation is proposed. The model carries some of the desirable features of the neurofuzzy approach, such as learning ability and good interpretability, while maintaining the merits of the COCOMO model. Unlike the standard neural network approach, this model is easily validated by experts and capable of generalization. In addition, it allows inputs to be continuous-rating values and linguistic values, therefore avoiding the problem of similar projects having different estimated costs. Also presented in this paper is a detailed learning algorithm. The validation, using industry project data, shows that the model greatly improves the estimation accuracy in comparison with the well-known COCOMO model.

1. Introduction Today, when software development has become an essential investment for many organizations, software engineering practitioners become more and more concerned about accurately predicting the cost and quality of a software product under development. Estimation accuracy is largely affected by modeling accuracy. Finding good models for software estimation is now one of the most important objectives of the software engineering community. Many software estimation models [1]-[4], [10] have been developed over the last decades. COCOMO is one of the most popular models. Many models cannot effectively cope with some realities of software engineering such as dealing with imprecise and uncertain information, and highly complex nonlinear relationships between variables. Soft computing provides software developers with some promising techniques such as fuzzy logic, artificial neural networks, and evolutionary computation for software modeling [5], [6]. In particular, fuzzy logic is powerful in solving real world applications with

Proceedings of the Third International Conference On Quality Software (QSIC’03) 0-7695-2015-4/03 $ 17.00 © 2003 IEEE

Danny Ho Toronto Design Center Motorola Canada Ltd. Markham, Ontario, L6G 1B3, Canada [email protected]

imprecise and uncertain information and in dealing with semantic knowledge [7]; and it is easily understood and interpreted. Artificial neural networks have the ability to model complex nonlinear relationships and are capable of approximating any measurable functions through learning. The neuro-fuzzy approach, by symbiotically integrating the merits of fuzzy logic and neural networks, provides a more powerful tool to solve many issues in software engineering, including software estimation. In this paper, we propose a novel neuro-fuzzy COCOMO model to estimate software development effort by combining neuro-fuzzy technique with the popular COCOMO model. The new model can be easily interpreted and generalized. Specifically, first the learning parameters in our model have concrete physical meaning and the entire decision process is clear to the users. As a result, our model can be interpreted and validated straightforwardly by experts, and overcomes the “black box” problem that is common in neural network approaches. Secondly, we choose triangular membership functions and use monotonic constraints for our model; consequently, our model is easily generalized. Another feature of our model is that it allows for continuous-rating values and therefore we can avoid the problem of similar projects with a large variance in cost estimations.

2. Neuro-fuzzy COCOMO model Our novel neuro-fuzzy model for software development effort estimation is shown in Figure 1. The inputs of this model are the software size and ratings of 22 cost drivers, including five scale factors (SFRi) and seventeen effort multipliers (EMRi). The output is the software development effort estimation. Ratings of cost drivers can be continuous numerical values, or linguistic terms such as “Low”, “Nominal” and “High”. The parameters in this model are calibrated by learning from industry project data. There are two major components in our neuro-fuzzy model:

zTwenty-two sub-models NFi: for each submodel, the input is the rating value of a cost driver, and the output is the corresponding multiplier value, which is used as the input of the COCOMO Model.

zCOCOMO Model [1], [2]: The inputs of the COCOMO Model are the size of software and the outputs of NFi, and the output is software effort estimation.

Size SFR1

NF1 SF1

SFR2

NF2

SF2

… SFR5

NF5

SF5

AuSize

Effort

17

¦ u EM i 1 i i1

NF6 …

EMR17

5

B0.01 SFi

EM1 EMR1

COCOMOII MODEL

EM17

NF22 Figure 1. Neuro-fuzzy model for software effort estimation

2.1. Structure of sub-model NFi There is a total of 22 cost drivers in our neuro-fuzzy model. Each cost driver represents one factor that contributes to the development effort, such as applications experience and product complexity. To evaluate the contribution, we use six qualitative rating levels. When expressed in linguistic terms, these six rating levels are Very Low (VL), Low (L), Nominal (N), High (H), Very High (VH) and Extra High (XH). Each rating level of every cost driver relates to a quantitative value used in the COCOMO model. Sub-model NFi is used to translate qualitative rating of a cost driver into a quantitative multiplier value and to calibrate these relations by industry project data. Note that not all six rating levels are valid for all cost drivers. A natural way to represent linguistic terms is to use fuzzy sets. In this paper, we define a fuzzy set for each linguistic term of every cost driver, i.e. “very low”, “low”, “nominal”, “high”, “very high”, “extra high”. We let the membership functions be triangular functions or other functions, and choose the universe of discourse to be the interval [1,6]. Roughly speaking, we use fuzzy numbers “about 1”, “about 2”,…, “about 6” to represent linguistic terms “very low”, “low”, “nominal”, “high”, “very high”, “extra high”, respectively.


We adopt Adaptive Neuro-Fuzzy Inference System (ANFIS) [8] for each NFi. We denote CDi SFi ,

i 1,2,,5 and CDi 5

EM i , i

1,2, ,17 . The

input of NFi is the rating value CDRik of the i-th cost driver CDi, and the output is the corresponding multiplier value, CDi. Figure 2 shows the ANFIS structure of submodel NFi for a 1-input 1-output system, which is functionally equivalent to a Takaki and Sugeno’s type of fuzzy system [9] having the rule base: Fuzzy Rule k: If CDRi is Aik, then fk=CDik , k=1,2,…,6 where the fuzzy set Aik represents a rating level ranging from “very low” to “extra high”, and CDik is the corresponding multiplier value of the cost driver CDi. Let P ik (x ) be the membership function of fuzzy set

Aik , k

1,2, ,6 . Thus the output of NFi is

CDi

¦w

k

k

where wk

fk

¦ w CD

Pik (x ) , wk

k

ik

k

wk . ¦ wj j

(1)

Layer1

Layer3

Layer2

w1

w1

A1

3

A2

3

N

…

…

…

A6

3

N

CDRi

N

w6

Layer4

Layer5

CDRi

w1CDi1 CDi

6

CDRi

…

w6CDi 6

w6

CDRi

Figure 2. ANFIS structure of sub-model NFi Theorem 1. If membership functions are the following triangular functions shown in Figure 3,

P ik ( x)

x (k 1) , ° ®(k 1) x , °0, ¯

k

1,2,,6

k 1 d x d k k d x d k 1, otherwise (2)

for any continuous rating value CDRi , the output

CDi of sub-model NFi is piecewise linear interpolation between the multiplier values CDi1 , CDi 2 , , CDi 6 of the i-th cost driver CDi , CDi CDik (CDik 1 CDik )(CDRi k ), if k d CDRi d k 1, k 1,2, ,6 (3)

Figure 3. Triangular membership functions

It is easy to see that each sub-model NFi gives the same results as linear interpolation in mathematics when we select triangular membership functions. This is a very intuitive interpretation for our neuro-fuzzy model. In addition, linear interpolation is easily generalized, and that is an important criterion for successful applications of neural networks and fuzzy logic techniques.


2.2. COCOMO model II COCOMO II Post Architecture Model is defined as: 5

Effort

A u Size

B 0.01u

17

¦ SFi u i 1 EM i

(4)

i 1

where A and B are baseline calibration constants, Size refers to the size of the software project measured in

terms of kSLOC (thousands of Source Lines of Code), SF is the scale factor, and EM is the effort multiplier.

From (6) and (7), we can rewrite our neuro-fuzzy model as follows:

Effort

f NF ( X , CD )

2.3. Learning algorithms

ª SF11 SF12 « SF SF22 21 « « « SF51 SF52 CD { « « EM 11 EM 12 « EM EM 22 21 « « « ¬ EM N 1 EM N 2 ª x1 º « x » « 2 » « » « » « x5 » X { « x6 » « x » « 7 » « » « x N1 » «x » ¬ N1 1 ¼

ª SFR1 º « SFR » 2 » « « » « » « SFR5 » « EMR1 » « EMR » 2 « » « » « EMRN » « Size » ¬ ¼

NN

(1) CDi1 d CDi 2 d CDi 3 d CDi 4 d CDi 5 d CDi 6

5

SFi

(6)

N

u EM i i 1

5

A u Size

B 0.01u

x

E dn is the actual effort for Project n, E n Effort n f NF ( X n , CD )

x

corresponding effort estimation by our neurofuzzy model, I INC (CD ) is the set of increasing cost drivers

x

whose higher rating values correspond to higher development effort, and I DEC (CD ) is the set of decreasing cost drivers

x

For our neuro-fuzzy model,

drivers for Project n, wn is the weight of Project n,

x

1,2, , N 1 , X

N1

(7)

¦ CDi u CD i 1 i i 6


(11)

X n is the software size and ratings of cost

(5)

i 1,2, , N 1

i 1

, (10)

i I INC (CD )

where x

( xi )CDik , ¦

(9)

subject to the following monotonic constraints:

k 1

A u Size

E dn

2

n

i I DEC (CD )

6

Effort

n

n 1

f NFi (CDi1 , CDi 2 , , CDi 6 )

B 0.01u

( X n , E dn ) ,

(2) CDi1 t CDi 2 t CDi 3 t CDi 4 t CDi 5 t CDi 6 ,

corresponding to the cost driver CDi , i

ik

1

¦ 2 w E

E

is the size of software and rating values of the cost drivers, and CD is the corresponding multiplier values of cost drivers. And if we choose triangular functions defined by Equation (2) to be the membership functions, substituting (2) into (1), we have

¦P

data points

CD can be formulated as the following optimizing problem:

17 , N 1 N 5 is the total number of cost drivers, xi is the rating value or linguistic term

project

n 1,2, , NN , the learning problem of parameters

SF16 º SF26 »» » » SF56 » , EM 16 » EM 26 » » » » EM N 6 ¼

where N

CDi

NN

Given

In this paper, we denote:

(8)

is

the

whose higher rating values correspond to lower development effort. The learning algorithm for our neuro-fuzzy model is as follows:

CDikl 1 where

CDikl D

D !0

wE wCDik wE n wCDik

wE wCDik

(12)

is the learning rate,

NN

¦ w E n

n

E dn

n 1

wE n wCDik

wCDi , wCDik for i 1,2, ,5

(13)

E n ln(Size)

(14)

wE n wCDik

wCDi wCDik

E n wCDi , CDi wCDik for i 6,7, , N 1 wf NFi P ik ( xi ) , wCDik for i 1,2, , N 1

(15)

(16)

2.4. Features of the neuro-fuzzy model The main characteristics of the neuro-fuzzy model put forward by this research are: x Interpretability: Although the neural network approach provides a powerful tool to model complex sets of relationships and learn from previous data, it has an inherent shortcoming: neither is it easy to understand nor is it to explain its decision process [7]. In the whole decision process, however, our neuro-fuzzy model is clear to users and its learning parameters EMi and SFi can be easily interpreted and validated by experts. In addition, if the rating value is a continuously variable value between 1 and 6, the model produces the same result as that using mathematical linear interpolation; consequently, our neuro-fuzzy model is more easily accepted for project management. x Continuous rating values: Our model allows for continuous rating values (e.g. between 1 and 6), a fact which avoids the problem of similar projects with large different effort estimations (e.g. extreme cases: 203 staff-months vs. 2886 staff-months). x Monotonic constraints: Monotonic constraints reflect expert knowledge of cost drivers. We use monotonic constraints to guarantee that the calibration results are reasonable. x Reduced number of learning parameters: The parameters of membership functions in our model are fixed. EMi and SFi are the only parameters to learn. x Knowledge integration: In our model, we can integrate expert knowledge with concrete project data by using fuzzy rules. x Good generalization: Our model has good generalization by using linear interpolation and imposing monotonic constraints. x Local learning: In our model, the learning models for the parameters are decoupled. This feature allows our model to learn just some of the parameters each time. In other words, our model can accumulate knowledge locally.


2.4.1. Ratings: Discrete Values Versus Continuous Values. The COCOMO model can only take on discrete ratings such as six linguistic terms: Very Low (VL), Low (L), Nominal (N), High (H), Very High (VH) and Extra High (XH), a constraint which might cause a problem in that the model might produce two rather different cost estimations for two similar projects. To illustrate this problem, let us consider two similar hypothetical projects P1 and P2: x For the cost drivers whose multiplier values are increasing, P1 is just below the lower limit of a linguistic value such as “high”, and P2 is just above this limit, and x For those whose multiplier values are decreasing, P1 is just above the lower limit of a linguistic value such as “high”, and P2 is just below this limit. If we select the limit between two contiguous multiplier values with largest difference for each driver, and P1 and P2 have the same nominal effort, say 100 staff-months, then for the COCOMO II.2000 model, the adjusted effort for P1 is 203 staff-months, but for P2 it is 2886 staff-months! So the difference between two estimations is over 14 times. The above problem results from using discrete rating values. Allowing continuously variable rating values as inputs, on the other hand, can easily eliminate this serious problem, due to its gradual transition. In our neuro-fuzzy model, we use fuzzy sets rather than discrete rating values to represent the linguistic terms (“very low”, “low”, etc) for all cost drivers, and use triangular member functions that have the desirable linear interpolation. As a result, our neuro-fuzzy model allows for continuous inputs and gives around 809 staffmonths for the above two projects P1 and P2. 2.4.2. Monotonic Constraints. Monotonic constraints are derived from expert knowledge and are critical for getting reasonable learning results. The following example illustrates their importance. The TURN cost driver represents the computer turnaround time. When the rating of TURN becomes higher, development effort should also increase. But if we do not use monotonic constraints on our model, we obtain the following fuzzy rules for the TURN cost driver: If TURN is “Very Low”, then f1=0.536 If TURN is “Low”, then f2=1.037 If TURN is “Nominal”, then f3=1 If TURN is “High”, then f4=1.105 If TURN is “Very High”, then f5=1.019 This result can be validated by experts. Obviously, the second and the fifth rules are not consistent. But when we place monotonic constraints on our model, we obtain the following reasonable rules:

If TURN is “Very Low”, then f1=0.592 If TURN is “Low”, then f2=0.989 If TURN is “Nominal”, then f3=1 If TURN is “High”, then f4=1.095 If TURN is “Very High”, then f5=1.095

3. Validation by industry project data In this section, we use industry project data to validate our neuro-fuzzy COCOMO model. There are a total of 69 project data available, including 6 project data from the industry [11], [12] and 63 project data from the original COCOMO’81 database [2]. We were not able to use the COCOMO II database due to absence of individual cost driver ratings for the project data. Because most of the project data are compatible only with the intermediate COCOMO’81 model, we will use the COCOMO’81 model for our validation. The COCOMO’81 model contains only 15 effort multipliers, yet our neuro-fuzzy COCOMO model is completely compatible with the COCOMO II model. Because outliers are abnormal project data with large noise, we need to preprocess the project data and remove 4 outliers (with project data deviating from over 50% of the COCOMO81 model output) from the original COCOMO’81 database. In other words, we use only 65 project data to train the neuro-fuzzy model, and use all 69 project data to test our neuro-fuzzy model. The detailed industrial project data are shown in Table 1. Because all these 6 projects used a later

COCOMO model which is slightly different from the original COCOMO’81 (e.g. three effort multipliers namely RUSE, VMVH and VMVT, are not used in the COCOMO’81 model), the cost estimation by using COCOMO’81 model must be adjusted by multiplying a constant of 0.94116, which reflects a common rating of RUSE(H), VMVH(L), and VMVT(L) for these projects. Note also that the actual effort refers to the development effort adjusted to be compatible with the definition of the COCOMO model.

3.1. Case I: Learning with all project data We use the 65 project data points to train the neurofuzzy model. The results are shown in Case I of Table 2 and Table 3. In Table 2, Relative Error,

RE

Effortest Effortact / Effortact , Effortact is

the actual effort, Effortest is the estimating effort; PERC is the percentage of projects that have a relative error under RE; IMPRV means improvement in comparison with estimation by the COCOMO model. If we consider projects with relative error within 20% of actual efforts, we notice that cost estimation accuracy, using neuro-fuzzy model has improved by about 15% when compared with the COCOMO’81 model. If we are concerned only with estimation accuracy for the industrial project data, there is a big improvement (referring to Table 3).

Table 1. Industrial project data Project No P1

Size (kESLOC ) 196.6

Effort (StaffMonth) 638.0

P2

51.8

185.0

P3

64.1

332.0

P4

130.0

619.9

P5

13.3

64.8

P6

19.9

76.6

High EMs

ACAP, PCAP, VEXP, LEXP, CPLX, TIME, STOR, TOOL, RUSE ACAP, PCAP, AEXP, VEXP, LEXP, CPLX, STOR, TOOL, RUSE VEXP, CPLX, STOR, TOOL, RUSE RELY, STOR, RUSE VEXP, RELY, CPLX, STOR, RUSE CPLX, TOOL, RUSE


Low EMs

Very Low EMs

DATA, VMVH, VMVT, SCED DATA, VMVH, VMVT, SCED DATA, VMVH, VMVT VEXP, LEXP, DATA, VMVH, VMVT VMVH, VMVT, MODP

TURN

AEXP, VEXP, LEXP, DATA, VMVH, VMVT

TURN

Table 2. Effort estimation for all 69 project data points RE

COCOMO81 Model PERC

20% 30%

71% 81%

Neuro-Fuzzy Model Case II Case III PER IMPR PER IMPR C V C V 88% 17% 88% 17% 92% 11% 92% 11%

Case I PERC IMPR V 86% 15% 92% 11%

Case IV PER IMPR C V 89% 18% 94% 13%

Table 3. Effort estimation for industrial project data Project No

Actual Effort

P1 P2 P3 P4 P5 P6

638.0 185.0 332.0 619.9 64.8 76.6

COCOMO81 Model Estimate Error 827.0 29% 152.2 -17% 279.8 -15% 701.4 13% 71.1 9% 83.1 8%

Case I Estimate Error 745.0 16% 167.3 -9% 322.0 -3% 651.5 5% 63.4 -2% 72.2 -5%

Neuro-Fuzzy Model Case II Case III Estimate Error Estimate Error 739.7 15% 728.6 14% 166.1 -10% 163.6 -11% 306.8 -7% 325.5 -1% 651.7 5% 642.0 3% 60.8 -6% 64.1 -1% 72.3 -5% 73.8 -3%

3.2. Case II: Learning with part of project data

4. Conclusions

In this case, we use 63 project data points to train our neuro-fuzzy model, saving two industrial project data points to test our model. The validation results are shown in Case II of Table 2 and Table 3. The neuro-fuzzy model also gives better cost estimation accuracy than the COCOMO’81 model.

We have proposed a novel neuro-fuzzy COCOMO model to estimate software development effort. By combining the neuro-fuzzy approach with the standard COCOMO models, we can take advantage of some of the desirable features of a neuro-fuzzy approach, such as learning ability and good interpretability. In addition, we choose fixed triangular membership functions and put monotonic constraints on our model. Consequently, our model is capable of generalization, an important criterion for successful applications of neural networks and fuzzy logic techniques. The learning parameters in our model have concrete physical meaning and the entire decision process is clear to the user; therefore this model can be interpreted and validated by experts. Another feature of our model is that it allows for continuous-rating values as inputs and therefore eliminates the problem of similar projects with large different cost estimations. Validation by industry project data shows that our model can greatly improve cost estimation accuracy when compared with the standard COCOMO model. Finally, the neuro-fuzzy technique allows the integration of numerical data and expert knowledge; it can be a powerful tool to tackle important problems in software engineering such as cost and quality prediction. Therefore, a promising line of future work is to extend the neuro-fuzzy approach to other cost and quality estimation models and tools such as COQUALMO, SLIM, SPR knowledgePLAN, and CA-Estimacs, among others.

3.3. Case III: Use larger weights for local organization data (industrial project data) In this case, we use the same 65 project data points to train our neuro-fuzzy model but place different weights on project data from the COCOMO’81 database (weights=1) and industrial project data (weights=2). The validation results are shown in Case III of Table 2 and Table 3. For local organization data, the results are better than Case I using same weights for all projects. Thus if we assign more weights on local data, we can get more accurate estimation for local project data.

3.4. Case IV: Without monotonic constraints In this case, we do not use monotonic constraints on multiplier values of cost drivers. The results are shown in Case IV of Table 2. We notice that the resulting estimating accuracy is higher in comparison with Case I (with monotonic constraints). However, some of the multiplier values learned are counter-intuitive. Therefore, we need to place monotonic constraints on our model.


References [1] B. Boehm et al., Software Cost Estimation with COCOMO II, Prentice Hall PTR, Upper Saddle River, New Jersey 07458, 2000. [2] B. Boehm, Software Engineering Economics, PrenticeHall, Inc., Englewood Cliffs, New Jersey 07632, 1981. [3] S. MacDonell and A. Gray, “A comparison of modeling techniques for software development effort prediction,” in Proceedings of the 1997 International Conference on Neural Information Processing and Intelligent Information Systems, Springer-Verlag, 1997, pp. 869–872. [4] S. Chulani, Bayesian Analysis of Software Cost and Quality Models, Ph.D. Dissertation, University of Southern California, 1999. [5] Y. Dote and S. J. Ovaska, “Industrial applications of soft computing: a review,” Proceedings of the IEEE, Vol. 89, No. 9, pp.1243-1265, September 2001. [6] S. Mitra, “Neuro-fuzzy rule generation: survey in soft computing framework,” IEEE Trans. on Neural Networks, Vol. 11, pp. 748–768, May 2000.


[7] R. Fuller. Introduction to Neuro-Fuzzy Systems, PhysicaVerlag, Heidelberg, 2000. [8] R. J. S. Jang, “ANFIS: adaptive-network-based fuzzy inference system,” IEEE Trans. Systems, Man, and Cybernetics, Vol. 23, pp. 665-685, 1993. [9] T. Takaki and M. Sugeno, “Fuzzy identification of systems and its application to modeling and control”, IEEE Trans. Syst., Man, Cybern., Vol. 15, pp.116-132, 1985. [10] M. Shepperd and G. Kadoda, “Comparing software prediction techniques using simulation”, IEEE Transactions on Software Engineering, Vol. 27, No. 11, pp. 1014-1022, November 1999. [11] D. Ho, “Experience report on COCOMO and the Costar tool from Nortel’s Toronto Laboratory,” in Eleventh International Forum on COCOMO and Software Cost Modeling, University of Southern California, Los Angeles, October 1996. [12] N. Panlilio-Yap and D. Ho, “Deploying software estimation technology and tools: the IBM SWS Toronto Lab experience,” in Ninth International Forum on COCOMO and Software Cost Modeling, University of Southern California, Los Angeles, October 1994.