IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2891
PAPER
Discrete Program-Size Dependent Software Reliability Assessment: Modeling, Estimation, and Goodness-of-Fit Comparisons Shinji INOUE†a) and Shigeru YAMADA† , Members
SUMMARY In this paper we propose a discrete program-size dependent software reliability growth model flexibly describing the software failure-occurrence phenomenon based on a discrete Weibull distribution. We also conduct model comparisons of our discrete SRGM with existing discrete SRGMs by using actual data sets. The program size is one of the important metrics of software complexity. It is known that flexible discrete software reliability growth modeling is difficult due to the mathematical manipulation under a conventional modeling-framework in which the time-dependent behavior of the cumulative number of detected faults is formulated by a difference equation. Our discrete SRGM is developed under an existing unified modeling-framework based on the concept of general order-statistics, and can incorporate the effect of the program size into software reliability assessment. Further, we discuss the method of parameter estimation, and derive software reliability assessment measures of our discrete SRGM. Finally, we show numerical examples of discrete software reliability analysis based on our discrete SRGM by using actual data. key words: software reliability assessment, modeling framework, program size, discrete Weibull distribution, heuristic parameter estimation algorithm, goodness-of-fit
1.
Introduction
Software reliability assessment in a testing-phase located in a final stage of the software development process is one of the important activities in developing a highly reliable software system. In the testing-phase, an implemented software system is tested to detect and correct software faults latent in the system. The software development manager has to assess the software reliability of the final software product especially in the final phase of the software development process to ship a reliable software system to the user. A software reliability growth model (SRGM) [1]–[4] is known as one of the useful mathematical tools to assess the software reliability quantitatively. However, as a role of software systems has been expanding rapidly, the size, complexity, and diversification of software systems have been growing drastically in recent years. Then we still need to investigate methodologies for developing more plausible SRGMs which enable us to assess the software reliability more accurately. For example, as one of the results of the research activities, generalization or unified approaches for software reliability growth modeling have been proposed based on nonhomogeneous Markov processes [2], [5], [6], order-statistics Manuscript received September 7, 2006. Manuscript revised March 19, 2007. Final manuscript received August 23, 2007. † The authors are with the Department of Social Systems Engineering, Faculty of Engineering, Tottori University, Tottori-shi, 680-8552 Japan. a) E-mail:
[email protected] DOI: 10.1093/ietfec/e90–a.12.2891
[7]–[10], an infinite server queueing theory [11], and so forth. These generalized and unified approaches provide us with new frameworks for software reliability growth modeling, and treat existing SRGMs proposed so far theoretically and generally. Then, the software development manager can develop a plausible SRGM which has good performance for actual software reliability assessment by using these modeling-frameworks. However, almost all of the unified or generalization approaches have been discussed for continuous-time software reliability growth modeling because SRGMs on the continuous-time domain is specifically applicable to the reliability analysis and better for mathematical manipulations. When the software development managers conduct software reliability assessment, they need to obtain software reliability data, such as software failure-occurrence times data or fault count data. The software failureoccurrence times data are obtained by observing the behavior of the executed software system in the testing-phase continuously, which are measured on the basis of CPU hours. For such software failure-occurrence times data based on the continuous-time domain, it would be better to use the continuous-time SRGM. However, In almost all of actual testing-phases, the frequency of the software failureoccurrence or the number of detected faults during each constant testing-period is frequently observed as software reliability data since obtaining the continuous-time data needs a lot of efforts. Such data collection activities yield the fault count data. When such data are obtained, it would be better to use discrete SRGM for software reliability modeling and analysis. Needless to say, the discrete SRGM also has proper characteristics for describing a software reliability growth process which depends on a unit of the software fault-detection period, such as the number of test runs, the number of executed test cases [12], [13]. Otherwise, as the topic of the discrete SRGM, SRGMs derived by discretization of the time-domain of continuous-time SRGMs have been proposed to solve some problems in parameter estimation of continuous-time SRGMs which are described by differential equations. Such discrete SRGMs are often called discretized SRGMs [14]. Considering that there are discrete SRGMs to describe software reliability growth processes depending on such discrete time-domain, we need to discuss a generalization or unified approach for plausible discrete software reliability growth modeling. Until now, a few generalization or unified approaches for discrete SRGMs have been discussed. Huang et al. [15] have discussed a
c 2007 The Institute of Electronics, Information and Communication Engineers Copyright
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2892
unified scheme of discrete nonhomogeneous Poisson process (NHPP) models by applying the concepts of weighted arithmetic, geometric, or harmonic means. And, Okamura et al. [16] have discussed a unified framework for discrete software reliability growth modeling based on the concept of the unification approach discussed by Langberg and Singpurwalla [7]. In recent years, a framework for discrete software reliability growth modeling with the effect of the program size which is one of the important metrics of software complexity has been proposed [17]. This paper focuses on flexible discrete software reliability growth modeling based on the concepts of the unification and generalization approaches mentioned above because software development managers have a great interest in a widely-applicable SRGM which can describe several types of software failure-occurrence phenomena flexibly. Especially in this paper, we develop a flexible discrete SRGM with the effect of the program size where the stochastic behavior of the software failure-occurrence times follows a discrete Weibull distribution. Our SRGM is based on the concept of the unified framework for discrete software reliability growth modeling with the effect of program size proposed by a literature [17]. And, several software reliability assessment measures which are useful metrics for quantitative software reliability assessment are also derived based on the concepts of the unified framework. Then, we compare the performance for software reliability assessment of our model with those of existing SRGMs. Finally we show numerical examples for software reliability analysis based on our model by using actual data. 2.
Unified Framework with Program Size
2.1 Modeling Framework We discuss a unified framework for discrete software reliability growth modeling with the effect of the program size on software reliability growth process [17]. The unified framework is based on the following basic assumptions from [16]: (A1) Whenever a software failure is observed, the fault which caused it will be detected immediately, and no new faults are introduced in the fault-detection procedure. (A2) Each software failure occurs at independently and identically distributed random times I with the discrete probability distribution P(i) ≡ Pr{I ≤ i} = ik=0 pI (k) (i = 0, 1, 2, · · · ), where pI (k) and Pr{A} represent the probability mass function for I and the probability of event A, respectively. (A3) The initial number of faults in the software system, N0 (> 0), is a random variable, and is finite. Now, let {N(i), i = 0, 1, · · · } denote a discrete stochastic process representing the number of faults detected up to i-th testing-period. From the assumptions above, the conditional probability that m faults are detected up to i-th testing-period given that N0 = n is derived as
Pr{N(i) = m | N0 = n} n = {P(i)}m {1 − P(i)}n−m . m
(1)
Then, we have the probability mass function that m faults are detected up to i-th testing-period as Pr{N(i) = m} n = {P(i)}m {1 − P(i)}n−m Pr{N0 = n} m n
(m = 0, 1, 2, · · · ).
(2)
The stochastic behavior of the software failureoccurrence phenomenon in the testing-phase can be characterized by giving a suitable probability mass function of the initial fault content N0 . In Eq. (2), we consider the case that the probability distribution of the initial fault content, N0 , follows a binomial distribution with parameters (K, λ) which is given as K n λ (1 − λ)K−n Pr{N0 = n} = n (0 < λ < 1 ; n = 0, 1, · · · , K).
(3)
Equation (3) has the following physical assumptions: (a) The software system consists of K lines of code (LOC) at the beginning of the testing-phase. (b) Each code has a fault with a constant probability λ. (c) Each software failure caused by a fault remaining in the software system occurs independently and randomly. These assumptions are useful to apply a binomial distribution to the probability mass function of the initial fault content in the software system, and to incorporate the effect of the program size into software reliability assessment [6], [17]. The program size is one of the important metrics of software complexity which influences the software reliability growth process in the testing-phase. Substituting Eq. (3) into Eq. (2), we can derive the probability mass function of the number of faults detected up to i-th testing-period as Pr{NB (i) = m} K n = {P(i)}m {1 − P(i)}n−m m n=m K n × λ (1 − λ)K−n n K = {λP(i)}m m K K−m × {λ(1 − P(i))}n−m (1 − λ)K−n n−m n=m K = {λP(i)}m {1 − λP(i)}K−m m (m = 0, 1, 2, · · · K).
(4)
INOUE and YAMADA: DISCRETE PROGRAM-SIZE DEPENDENT SOFTWARE RELIABILITY ASSESSMENT
2893
From Eq. (4), several types of discrete SRGMs with the effect of program size can be developed by giving suitable probability distributions for the software failure-occurrence times, respectively. We should mention that Eq. (4) is essentially same as a general order statistics model discussed in the literatures [7]–[10] when the parameters K → ∞ and λ → 0.
generalized software reliability assessment measures in Eqs. (5)–(9) by giving the suitable probability mass function of the initial fault content, N0 , and the software failureoccurrence times distribution, P(i). 3.
Discrete Weibull-Type SRGM
3.1 Modeling 2.2 Reliability Assessment Measures Reliability assessment measures, such as the expectation and variance of the number of detected faults, software reliability function, instantaneous and cumulative mean time between software failures (MTBFs), are well-known metrics for quantitative software reliability assessment. Inoue and Yamada [17] derived generalized software reliability assessment measures under the basic assumptions on the software failure-occurrence phenomenon in Eq. (2). The expectation of the number of detected faults, E[N(i)], is derived as n n z {P(i)}z {1 − P(i)}n−z E[N(i)] = z n
z=0
· Pr{N0 = n} = E[N0 ]P(i).
β
(5)
And its variance, Var[N(i)], is also derived as Var[N(i)] = E[N(i)2 ] − (E[N(i)])2 = Var[N0 ]{P(i)}2 + E[N0 ]P(i){1 − P(i)}. (6) A discrete software reliability function is defined as the probability that a software failure does not occur in the timeinterval (i, i + h](i, h = 0, 1, 2, · · · ) given that the testing or the operation has been going up to i-th testing-period [12]. Then, the discrete software reliability function, R(i, h), under the basic assumption in Eq. (2) is derived as R(i, h) = Pr{N(i + h) = k | N(i) = k}Pr{N(i) = k} k
=
k
·
{P(i)}k {1 − P(i + h)}−k
n n
k
{1 − P(i + h)}n · Pr{N0 = n} .
Developing a widely-applicable SRGM which can describe the software failure-occurrence phenomenon flexibly and conserve a certain level of accuracy for software reliability assessment is better to use in project management of an actual software development. In this paper, we develop a flexible discrete SRGM with the effect of the program size under the modeling framework in Eq. (4). For flexible discrete software reliability growth modeling in this paper, we apply a discrete Weibull distribution [18], [19] to the software failure-occurrence times distribution. That is, we assume that the software failure-occurrence phenomenon in the testing-phase is described as Weibull order statistics essentially. The probability distribution function of the discrete Weibull distribution is given as
(7)
Further, discrete instantaneous and cumulative MTBFs, MTBF I (i) and MTBFC (i), are also derived as (8) MTBF I (i) = 1 (E[N(i + 1)] − E[N(i)]), (9) MTBFC (i) = i E[N(i)], respectively. Specific software reliability assessment measures which are applicable to actual software reliability assessment in a specific testing-phase can be derived from the
P(i) = 1 − (1 − p)i ,
(10)
where p represents the probability that a software failure caused by a fault is observed per one testing-period, and β the shape parameter. And its hazard rate function, z(i), is derived as z(i) = 1 − (1 − p)(i+1)
β
−iβ
.
(11)
The discrete Weibull distribution subsumes geometric and Rayleigh distribution as the special cases. In the case that β = 1 and β = 2, the discrete Weibull distribution can be regarded as the geometric distribution and the Rayleigh distribution, respectively. Further, from Eq. (11), we can see that the discrete Weibull distribution can describe flexibly the stochastic behavior of the software failure-occurrence times by the shape parameter β. That is, the discrete Weibull distribution indicates a decreasing software failure rate (DFR) for 0 < β < 1, a constant software failure rate (CFR) for β = 1, and a increasing software failure rate (IFR) for β > 1, which are useful properties for describing the software failure-occurrence phenomenon in the testingphase flexibly. As to software reliability assessment measures, the expectation and variance of the number of detected faults are derived as β
E[NB (i)] = Kλ{1 − (1 − p)i },
(12)
iβ
Var[NB (i)] = Kλ{1 − (1 − p) } β · 1 − λ{1 − (1 − p)i } ,
(13)
from Eqs. (5) and (6), respectively. The discrete software reliability function is derived as
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2894
K β β RB (i, h) = 1 − λ{(1 − p)i − (1 − p)(i+h) } ,
(14)
from Eq. (7). And, the instantaneous and cumulative MTBFs are also derived as β β MTBF I,B (i) = 1 Kλ{(1 − p)i − (1 − p)(i+h) } , (15) and β MTBFC,B (i) = i Kλ{1 − (1 − p)i } ,
(16)
from Eqs. (8) and (9), respectively. 3.2 Parameter Estimation We also discuss a method of parameter estimation for our SRGM. As to our discrete Weibull-type SRGM, we need to estimate the parameters λ, p, and β by using fault count data. Ordinarily, the method of maximum-likelihood based on the stochastic properties of an SRGM is applied to get parameter estimates. However, it is very difficult to obtain the parameter estimates, λ, p, and β, of the parameters λ, p, and β of our SRGM simultaneously by using the ordinary maximum-likelihood estimation method. For estimating the parameters λ and p in our SRGM, we have to set or estimate the value of parameter β in advance, which characterizes the type of stochastic behavior of the probability distribution function of the software failure-occurrence times distribution.
Then, we use a heuristic parameter estimation algorithm accompanying the method of maximum-likelihood to obtain the parameter estimates simultaneously. Our heuristic parameter estimation algorithm is based on the basic notion of the Nelder-Mead optimization method [20]. Such parameter estimation algorithm was also used for a generalized-gamma software reliability model proposed by Okamura et al. [21]. Figure 1 shows our heuristic parameter estimation algorithm. In our algorithm shown in Fig. 1, we set values of the shape parameter β1,1 , the variation rate r, and the initial variation width z1 . After the setting, we estimate the parameters λ and p with respect to each D, such as βm,l −zl , βm,l βm,l +zl , by using the method of maximum-likelihood, respectively. As to the estimation of parameters λ and p by using the method of maximum-likelihood in our algorithm, the logarithmic likelihood function can be derived by the following mathematical manipulation. Suppose that we have observed N data pairs (ti , yi )(i = 0, 1, 2, · · · , N ; t0 < t1 < · · · < tN ) with respect to the total number of faults, yk , detected during a constant time-interval (0, tk ]. Then, the likelihood function, l, for the discrete stochastic process NB (i) in Eq. (4) can be derived as l = Pr{NB (t1 ) = y1 , NB (t2 ) = y2 , · · · , NB (tN ) = yN } =
N
Pr{NB (ti ) = yi | NB (ti−1 ) = yi−1 }
i=1
· Pr{NB (t1 ) = y1 }, (17) by using the Bayes’ formula and a Markov property [22]– [24]. The conditional probability in Eq. (17), Pr{NB (ti ) = yi | NB (ti−1 ) = yi−1 }, can be derived as Pr{NB (ti ) = yi | NB (ti−1 ) = yi−1 } K − yi−1 = {z(ti−1 , ti )}yi −yi−1 yi − yi−1 {1 − z(ti−1 , ti )}K−yi ,
(18)
where z(ti−1 , ti ) =
λ{P(ti ) − P(ti−1 )} . 1 − λP(ti−1 )
(19)
Then, we can rewrite Eq. (17) as N K − yi−1 l= {z(ti−1 , ti )}yi −yi−1 yi − yi−1 i=1
{1 − z(ti−1 , ti )}K−yi ,
(20)
by using Eq. (18). In Eqs. (17)–(20), t0 = 0, y0 = 0, and P(t0 ) = 0. Accordingly, the logarithmic likelihood function can be derived as log l ≡ L = log K! − log{(K − yN )!} − Fig. 1
Heuristic parameter estimation algorithm.
N i=1
log{(yi − yi−1 )!} + yN log λ
INOUE and YAMADA: DISCRETE PROGRAM-SIZE DEPENDENT SOFTWARE RELIABILITY ASSESSMENT
2895
+
N
(yi − yi−1 ) log{P(ti ) − P(ti−1 )}
i=1
+ (K − yN ) log{1 − λP(ti )}.
(21)
By using Eq. (21), we can estimate the parameters λ and p if we can set the values of the program size K and the parameter β. The simultaneous logarithmic likelihood equations with respect to the parameters λ and p can be derived as ∂L yN = + (K − yN ) ∂λ λ 1
β
· {(1 − p)tN − 1}
β
1 − λ{1 − (1 − p)tN }
= 0,
yi − yi−1 ∂L = β β ∂p i=1 {(1 − p)ti−1 − (1 − p)ti } N
β
β
(K − yN ){tβN λ(1 − p)tN −1 } = 0, − 1 − λ{1 − (1 − p)tN }
(23)
respectively. Solving Eq. (22) with respect to λ, we can obtain yN λ= . (24) β K{1 − (1 − p)tN } Substituting Eq. (24) into Eq. (23), we can obtain the following equation: tβN −1
tβN yN (1 − p)
β
i=1 β
1 {(1 − p)
β
ti−1
β
− (1 − p)ti } β
β (1 − p)ti−1 −1 }. · {tiβ (1 − p)ti −1 − ti−1
(25)
Accordingly, we can obtain the maximum-likelihood estimates λ and p, respectively, by solving the simultaneous likelihood functions in Eqs. (24) and (25) numerically. After getting the parameter estimates, such as λ and p, and calculating the value of the maximum likelihood with respect to each D, we continue the iteration procedure denoted by l in Fig. 1 until “Condition 1” is satisfied. If “Condition 1” is satisfied, then we examine whether “Condition 2” is satisfied or not, where L(β∗0 ) = 0. If “Condition 2” is not satisfied, then we reduce the searching width of the parameter β from zm to zm+1 according to the variation rate r, and go back to “Step 1” along with the iteration procedure denoted by m. Following the above iteration procedures as shown in Fig. 1, we can obtain parameter estimates λ, p, and β consequently. 4.
G(n) = ka(1+δ log b) (k > 0, 0 < a < 1, 0 < b < 1 ; n = 0, 1, 2, · · · ), (26) where k represents the initial fault content, δ the constant time-interval, a and b the constant parameters. These parameter estimates in Eq. (26) are estimated by regression analysis.
β (1 − p)ti−1 −1 } · {tiβ (1 − p)ti −1 − ti−1
1 − (1 − p)tN N = (yi − yi−1 )
• Discrete Gompertz Curve Model. Let G(n) be the cumulative number of faults detected up to n-th testingperiod. Then, n
(22)
β
ment of our discrete Weibull-type SRGM with that of existing discrete SRGMs, such as a discrete Gompertz curve model (D-GOMP) [25], [26], a discrete logistic curve models (D-LOGI) [26], [27], a geometric error detection rate model (D-GEDR) [12], and continuous-time SRGMs, such as an exponential SRGM (C-EXPO) [28] and a delayed Sshaped SRGM (C-DELAYS) [29]. In the following, we show the details of these SRGMs, respectively.
Model Comparisons
We compare the performance for software reliability assess-
• Discrete Logistic Curve Model. Let L(n) be the cumulative number of faults detected up to n-th testingperiod. Then, L(n) = k {1 + m(1 − δα)n } (k > 0, α > 0, m > 0 ; n = 0, 1, 2, · · · ),
(27)
where k represents the initial fault content, δ the constant time-interval, α and m the constant parameters. These parameter estimates in Eq. (27) are estimated by regression analysis. • Geometric Error Detection Rate Model. This discrete SRGM is one of the discrete exponential NHPP models, and is developed by assuming that the expected number of errors detected per testing-period is geometrically decreased. Let E(n) be the expected number of faults detected up to n-th testing-period. Then, D(1 − rn ) (1 − r) (D > 0, 0 < r < 1 ; n = 0, 1, 2, · · · ),
E(n) =
(28)
where D represents the expected initial number of faults detected by the first testing-period, r the decreasing ratio for the expected number of faults detected per one testing-period. These parameter estimates in Eq. (28) are estimated by a method of maximumlikelihood based on the properties of the NHPP. • Exponential SRGM. This SRGM is one of the classical continuous-time NHPP models, but is still wellapplied to practical software reliability assessment. Let H(t) be the expected number of faults detected up to testing-time t. Then,
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2896
H(t) = a(1 − e−bt )
Fig. 2
Estimated numbers of detected faults for DS1.
Fig. 3
Estimated numbers of detected faults for DS2.
(a > 0, b > 0),
(29)
where a represents the expected total number of potential faults detected in an infinitely long duration, and b the fault-detection rate. These parameter estimates in Eq. (29) are estimated by the method of maximumlikelihood based on the NHPP. • Delayed S-shaped SRGM. This continuous-time NHPP model is also well-applied to practical software reliability assessment, and indicates an S-shaped software reliability growth curve. Let M(t) be the expected number of faults detected up to testing-time t. Then, M(t) = a 1 − (1 + bt)e−bt (a > 0, b > 0),
(30)
where a represents the expected initial fault content, and b the failure-occurrence rate or the fault-detection rate. These parameter estimates in Eq. (30) are also estimated by the method of maximum-likelihood based on the NHPP. The discrete Gompertz and logistic curve models are deterministic S-shaped models which are called discrete Sshaped statistical data analysis (SDA) models. These SDA models were developed by discretizing the original continuous-time SDA models based on Hirota’s bilinearization methods [30]. In this paper, we arrange the following four actual data sets to be used in the model comparisons: DS1 : (ti , yi )(i = 1, 2, · · · , 22 ; t22 = 22, y22 = 212) where ti is measured on the basis of weeks and the program size K = 1.630 × 105 (LOC) [31],
INOUE and YAMADA: DISCRETE PROGRAM-SIZE DEPENDENT SOFTWARE RELIABILITY ASSESSMENT
2897
Fig. 4
Estimated numbers of detected faults for DS3.
Fig. 5
Estimated numbers of detected faults for DS4.
DS2 : (ti , yi )(i = 1, 2, · · · , 25 ; t25 = 25, y25 = 136) where ti is measured on the basis of CPU hours and the program size K = 2.170 × 104 (LOC) [32], DS3 : (ti , yi )(i = 1, 2, · · · , 24 ; t24 = 24, y24 = 296) where ti is measured on the basis of weeks and the program size K = 1.972 × 105 (LOC) [31], DS4 : (ti , yi )(i = 1, 2, · · · , 35 ; t35 = 35, y35 = 1301) where ti is measured on the basis of months and the program size K = 1.240 × 105 (LOC) [33]. Among these actual data sets, DS1 and DS2 indicate exponential software reliability growth curves, and the remainder indicates S-shaped ones, respectively. To start with, Figs. 2–5 depict the time-dependent behavior of the estimated numbers of detected faults based on our model and the four discrete SRGMs for each actual data sets arranged in this paper. In these figures, the dotted
lines with black square symbols represent the actual data, the solid lines the time-dependent behavior of the estimated numbers of faults based on our model, and the other dotted lines based on the existing discrete SRGMs cited in this paper. From these figures, we see that the estimated numbers of detected faults differ from the discrete SRGM to the others. This implies the importance of model selection activities for accurate software reliability assessment. And we can say that our model fits better to the actual data sets indicating both exponential and S-shaped software reliability growth curves. For quantitative comparisons on the performance of these models, we use the following criteria: a root mean squared error (RMSE) [34], a mean magnitude of relative error (MMRE) [34], and predicted relative errors [2]. The RMSE and MMRE are calculated as
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2898 Table 1 DS1 DS2 DS3 DS4
Our model 5.097 2.627 5.918 30.29
DS1 DS2 DS3 DS4
Our model (×10−1 ) 0.644 0.206 0.814 1.030
Table 2
Results of model comparisons based on the RMSE. D-GOMP 27.29 8.164 8.387 169.0
D-LOGI 33.33 9.336 18.62 221.8
N 1 [yk − E[N(tk )]2 , N k=1
N 1
yk − E[N(tk )]
MMRE =
, N yk
D-LOGI (×10−1 ) 2.811 0.859 1.110 3.506
(31)
(32)
k=1
respectively. The RMSE represents the mean error of the number of detected fault between the estimated and actual ones. And the MMRE also represents the mean magnitude of error between the estimated and actual cumulative number of detected faults, which is relative to the actual cumulative number of detected faults at each testingtime. These criteria indicate how well each model represents to the actual fault-detection behavior. From Eqs. (31) and (32), we can see that smaller values of the RMSE and MMRE indicates smaller fitting errors, respectively, thus better goodness-of-fit. The predicted relative error, PRE[te ], is calculated as PRE[te ] =
y(te ; tq ) − nq , nq
(33)
where y(te ; tq ) is the estimated value of the number of detected faults at the testing-termination time, tq , by using the observed data collected up to the arbitrary testing-time, te (0 < te ≤ tq ), and nq the observed number of faults detected up to the termination time of the testing. Then, we can say that the SRGM whose value of the predicted relative errors at each testing-period or testing-time is closed to zero has better performance on the predictive accuracy. Tables 1 and 2 show the results of model comparisons based on the criteria on the RMSE and MMRE, respectively. In both Tables 1 and 2, the values of RMSE and MMRE of the geometric error detection rate model are the same as that of the exponential SRGM even though the testing time-domain of these SRGM are different each other. This is caused by the model structure of these two models. That is, rn in Eq. (28) and e−bt in Eq. (29) can be expanded as rn ≡ (1 − δb)n = 1 − nbδ + o(δ2 ) + · · · , e
−bt
≡e
−δbn
= 1 − nbδ + o(δ ) + · · · , 2
C-EXPO 8.549 5.960 24.56 113.8
C-DELAYS 6.053 12.95 12.25 64.96
Results of model comparisons based on the MMRE. D-GOMP (×10−1 ) 2.271 0.740 0.575 2.625
RMSE =
D-GEDR 8.549 5.960 24.56 113.8
(34) (35)
respectively, where t = nδ. Thus, Eq. (28) is the sames as
D-GEDR (×10−1 ) 1.827 0.689 3.050 5.368
C-EXPO (×10−1 ) 1.827 0.687 3.050 5.368
C-DELAYS (×10−1 ) 0.892 1.551 1.023 1.124
Eq. (29) up to the first order of δ. This means that these two models, the geometric error detection rate model and the exponential SRGM, have essentially the same properties when the estimates of D/(1−r) in Eq. (28) is the same as that of a in Eq. (29). In Table 2 we can see that our model does not have the best performance on the MMRE for DS3. This result is influenced by the fact that the discrete Gompertz curve model has a higher accuracy of estimating the number of detected faults only during the early stage of the testing in DS3 compared with our model. However, from Tables 1 and 2, except for the model comparison in DS3, we can say that our model has the best performance on the RMSE and MMRE among the other SRGMs cited in this paper. And we can also say that our model conserves a certain level of goodness-of-fit to the actual data sets cited in this paper. Additionally, Figs. 6–9 show the predicted relative errors of our model and each SRGMs cited in this paper for DS1–DS4, respectively. For each data set, each SRGM has 17 points of the testing-time, te ’s, which are calculated as te =[the termination time of the testing × the ratio of the testing-progress],
(36)
respectively, where [x] represents the Gaussian symbol for any real number x and the ratio of the testing-progress in Eq. (36) takes values from 0.2 to 1.0 with the intervals of 0.05. The solid lines in Figs. 6–9 represent the predictive performance of our model and the dotted lines the performance of the other existing SRGMs, respectively. Form Figs. 6–9, especially after 60% or 70% of the ratio of the testing-progress, we can say that our model has good predictive performance for the actual data sets indicating both exponential and S-shaped software reliability growth curves. The predicted relative errors of the discrete Gompertz and logistic curve models indicate stable behavior even in the early stage of testing but the predictive accuracy does not always have the best performance over all the data sets. And we can also say that the predictive accuracy of the geometric error detection rate model and the exponential SRGM have better predictive performance for the actual data sets DS2 indicating exponential software reliability growth curves, but not for the data indicating S-shaped reliability growth curves.
INOUE and YAMADA: DISCRETE PROGRAM-SIZE DEPENDENT SOFTWARE RELIABILITY ASSESSMENT
2899
5.
Fig. 6
Predicted relative errors for DS1.
Fig. 7
Predicted relative errors for DS2.
Numerical Examples for Our Model
We show examples for software reliability analysis based on our model by using the actual data sets of DS4. Based on the parameter estimation algorithm discussed in Sect. 3, the parameter estimates of λ, p, and β for DS4 are obtained as p = 2.631 × 10−3 , and β = 2.065, respecλ = 1.067 × 10−2 , tively. Then, the expected number of initial fault content can be estimated as K × λ 1323. Figure 10 depicts the estimated expected number of detected faults, E[NB (i)], and its 95% confidence limits. The 100γ% confidence limits for E[NB (i)] are derived as B (i)], (37) E[NB (i)] ± Kγ Var[N
where Kγ indicates the 100(1 + γ)/2 percent point of the standard normal distribution [1]. Deriving the confidence limits of the expected number of detected faults is meaningful for software quality control activities in the testingphase. Figure 11 shows the estimated discrete software reliB (45, h), which is assuming that the testing is conability, R ducted until the 45-th testing-period. Assuming that the testing is conducted until the 45-th testing-period and the user of this software operates under the same environment as the B (45, 10), testing, we can estimate the software reliability, R to be about 0.252. Additionally, Fig. 12 depicts the timedependent behavior of the estimated instantaneous MTBF, MT BF I,B (i), for DS4. From Fig. 12, we see that the estimated instantaneous MTBF decreases in the early stage of
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2900
Fig. 8
Predicted relative errors for DS3.
Fig. 9
Predicted relative errors for DS4.
the testing, but then the MTBF increases as the testing-times goes on. An increase of MTBF means growing of the software reliability. The instantaneous MTBF at the termina tion time of the testing is estimated as MT BF I (35) 0.204 (month). 6.
Concluding Remarks
We proposed a flexible discrete software reliability growth model which is based on a framework for discrete software reliability growth modeling with the effect of program size, where the software failure-occurrence times distribution follows a discrete Weibull distribution. And we discussed its parameter estimation based on a heuristic algorithm. Then, we conducted comparisons of the performance of our discrete model with that of existing discrete SRGMs in terms
of several criteria. Ordinarily, shapes of software reliability growth curves indicated by fault count data collected in actual testingphases are specified into S-shaped and exponential. Then, our flexible discrete SRGM might contribute to reduction of labor for selecting a discrete SRGM having better or the best performance for software reliability assessment whichever shape the software reliability growth curve indicates. Additionally, our flexible discrete SRGM enables us to assess the software reliability with the effect of the program size which is one of the metrics of software complexity influencing on the software reliability growth process. Developing flexible discrete SRGM has been difficult under a conventional modeling-framework in which the time-dependent behavior of the cumulative number of detected faults is formulated by a difference equation. However, we overcame the above
INOUE and YAMADA: DISCRETE PROGRAM-SIZE DEPENDENT SOFTWARE RELIABILITY ASSESSMENT
2901
Fig. 10 (DS4).
Estimated expected number of detected faults, E[NB (i)], and its 95% confidence limits.
In the future studies, we have to investigate more the usefulness and the validity of our flexible discrete SRGM by using many actual data sets. And we also have to research the appropriate range of program size to bring out more the feature of our model because the stochastic property on the software failure-occurrence phenomenon formulated by our model, a binomial distribution, can be regarded as a Poisson distribution as the parameters K and λ, respectively, tend to the infinite and zero. Acknowledgments
Fig. 11
B (45, h). (DS4) Estimated discrete software reliability, R
This work was supported in part by the Grant-in-Aid for Young Scientists (B), Grant No. 19710129, and the Grantin-Aid for Scientific Research (C), Grant No. 18510124, from the Ministry of Education, Sports, Science, and Technology of Japan. This work was also conducted as a part of the 2007 Research and Development Project of the Tottori University Venture Business Laboratory. References
Fig. 12
Estimated instantaneous MTBF, MT BF I,B (i). (DS4)
issue by applying the framework with the effect of the program size discussed in Sect. 2 to discrete software reliability growth modeling.
[1] S. Yamada and S. Osaki, “Software reliability growth modeling: Models and applications,” IEEE Trans. Softw. Eng., vol.SE-11, no.12, pp.1431–1437, 1985. [2] J.D. Musa, D. Iannio, and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill, New York, 1987. [3] H. Pham, Software Reliability, Springer-Verlag, Singapore, 2000. [4] S. Yamada, “Software reliability models,” in Stochastic Models in Reliability and Maintenance, ed. S. Osaki, pp.253–280, SpringerVerlag, Berlin, 2002. [5] J.G. Shanthikumar, “A general software reliability model for performance prediction,” Microelectronics and Reliability, vol.21, no.5, pp.671–682, 1981. [6] M. Kimura, S. Yamada, H. Tanaka, and S. Osaki, “Software reliability measurement with prior-information on initial fault content,” Trans. IPSJ, vol.34, no.7, pp.1601–1609, 1993. [7] N. Langberg and N.D. Singpurwalla, “A unification of some software reliability models,” SIAM J. Scientific and Statistical Comput-
IEICE TRANS. FUNDAMENTALS, VOL.E90–A, NO.12 DECEMBER 2007
2902
ing, vol.6, no.3, pp.781–790, 1985. [8] D.R. Miller, “Exponential order statistic models of software reliability growth,” IEEE Trans. Softw. Eng., vol.SE-12, no.1, pp.12–24, 1986. [9] A.E. Raftery, “Inference and prediction for a general order statistic model with unknown population size,” J. American Statistical Association, vol.82, no.400, pp.1163–1169, 1987. [10] H. Joe, “Statistical inference for general-order-statistics and nonhomogeneous-Poisson-process software reliability models,” IEEE Trans. Softw. Eng., vol.15, no.11, pp.1485–1490, 1989. [11] T. Dohi, T. Matsuoka, and S. Osaki, “An infinite server queueing model for assessment of the software reliability,” Electron. Commun. Jpn. 3, Fundam. Electron. Sci, vol.85, no.3, pp.536–544, 2000. [12] S. Yamada and S. Osaki, “Discrete software reliability growth models,” J. Applied Stochastic Models and Data Analysis, vol.1, no.1, pp.65–77, 1985. [13] T. Kitaoka, S. Yamada, and S. Osaki, “A discrete non-homogeneous error detection rate model for software reliability,” IECE Trans., vol.E69, no.8, pp.859–865, Aug. 1986. [14] S. Inoue and S. Yamada, “Discrete software reliability assessment with discretized NHPP models,” Computers & Mathematics with Applications: An Intern. J., vol.51, no.2, pp.161–170, 2006. [15] C.Y. Huang, M.R. Lyu, and S.Y. Kuo, “A unified scheme of some nonhomogeneous Poisson process models for software reliability estimation,” IEEE Trans. Softw. Eng., vol.29, no.3, pp.261–269, 2003. [16] H. Okamura, A. Murayama, and T. Dohi, “EM algorithm for discrete software reliability models: A unified parameter estimation method,” Proc. 8th IEEE International Symposium on High Assurance Systems Engineering (HASE’04), pp.219–228, 2004. [17] S. Inoue and S. Yamada, “Generalized discrete software reliability modeling with effect of program size,” IEEE Trans. Syst. Man Cybern. A, Syst. Humans, vol.37, no.2, pp.170–179, 2007. [18] T. Nakagawa and S. Osaki, “The discrete Weibull distribution,” IEEE Trans. Reliab., vol.R-24, no.5, pp.300–301, 1975. [19] J.D. Kalbfleisch and R.L. Prentice, The Statistical Analysis of Failure Time Data, Second ed., John Wiley & Sons, Hoboken, New Jersey, 2002. [20] J.A. Nelder and R. Mead, “A simplex method for function minimization,” Comput. J., vol.7, pp.308–313, 1969. [21] H. Okamura, M. Ando, and T. Dohi, “Generalized-gamma software reliability model,” IEICE Trans. Inf. & Syst. (Japanese Edition), vol.J87-D-I, no.8, pp.805–814, Aug. 2004. [22] S. Osaki, Applied Stochastic System Modeling, Springer-Verlag, Berlin, Heidelberg, 1992. [23] S.M. Ross, Introduction to Probability Models, Sixth ed., Academic Press, San Diego, 1997. [24] K.S. Trivedi, Probability and Statistics with Reliability, Queueing and Computer Science, Second ed., John Wiley & Sons, New York, 2002. [25] D. Satoh, “A discrete Gompertz equation and a software reliability growth model,” IEICE Trans. Inf. & Syst., vol.E83-D, no.7, pp.1508–1513, July 2000. [26] D. Satoh and S. Yamada, “Discrete equations and software reliability growth models,” Proc. 12th IEEE International Symposium on Software Reliability Engineering (ISSRE’01), pp.176–184, 2001. [27] D. Satoh and S. Yamada, “Parameter estimation of discrete logistic curve models for software reliability assessment,” Japan J. Industrial and Applied Mathematics, vol.19, no.1, pp.39–54, 2002. [28] A.L. Goel and K. Okumoto, “Time-dependent error-detection rate model for software reliability and other performance measures,” IEEE Trans. Reliab., vol.R-28, no.3, pp.206–211, 1979. [29] S. Yamada, M. Ohba, and S. Osaki, “S-shaped reliability growth modeling for software error detection,” IEEE Trans. Reliab., vol.R32, no.5, pp.475–484, 1983. [30] R. Hirota, “Nonlinear partial difference equations, V. Nonlinear equations reducible to linear equations,” J. Physical Society of Japan, vol.46, no.1, pp.312–319, 1979.
[31] T. Fujiwara and S. Yamada, “A new testing-path coverage measure — Testing-domain metrics based on a software reliability growth model,” Proc. 13th IEEE International Symposium on Software Reliability Engineering (ISSRE’02), pp.71–75, 2002. [32] A.L. Goel, “Software reliability models: Assumptions, limitations, and applicability,” IEEE Trans. Softw. Eng., vol.SE-11, no.12, pp.1411–1423, 1985. [33] W.D. Brooks and R.W. Motley, “Analysis of discrete software reliability models,” Technical Report, RADC-TR-80–84, Rome Air Development Center, New York, 1980. [34] J.J. Dolado, “A validation of the component-based method for software size estimation,” IEEE Trans. Softw. Eng., vol.26, no.10, pp.1006–1021, 2000.
Shinji Inoue was born in Fukuoka Prefecture, Japan, on April 6, 1978. He received the B.S.E., M.S., and Ph.D. degrees from Tottori University in 2001, 2003, and 2006, respectively. Since 2006, He has been working as Assistant Professor at the Faculty of Engineering of Tottori University, Japan. His research interests include software reliability engineering, quality control, and project management. He is a regular member of the IEEE, ORSJ, JSQC, IPSJ, SPM. Dr. Inoue was a winner of the The IEEE Reliability Society Japan Chapter, 2006 Outstanding Young Scientist Award.
Shigeru Yamada was born in Hiroshima Prefecture, Japan, on July 6, 1952. He received the B.S.E., M.S., and Ph.D. degrees from Hiroshima University in 1975, 1977, and 1985, respectively. From 1977 to 1980, he worked at the Quality Assurance Department of Nippondenso Co., Aichi Prefecture, Japan. From 1983 to 1988, he was an assistant professor of Okayama University of Science, Okayama, Japan. From 1988 to 1993, he was an associate professor at the Faculty of Engineering of Hiroshima University, Japan. Since 1993, he has been working as a professor at the Faculty of Engineering of Tottori University, Tottori, Japan. He has published numerous technical papers in the areas of software reliability models, project management, reliability engineering, and quality control. He has authored several books entitled, such as Software Reliability: Theory and Practical Application (Soft Research Center, Japan, 1990), Introduction to Software Management Model (Kyoritsu Shuppan, Tokyo, 1993), Software Reliability Models: Fundamentals and Applications (JUSE, Tokyo, 1994), Statistical Quality Control for TQM (Corona Publishing, Tokyo, 1998), and Software Reliability: Model, Tool, Management (The Society of Project Management, Narashino, Japan, 2004). Dr. Yamada received the Best Author Award from the Information Processing Society of Japan in 1992, the TELECOM System Technology Award from the Telecommunications Advancement Foundation in 1993, the Best Paper Award from the Reliability Engineering Association of Japan in 1999, International Leadership Award in Reliability Engg. Research from the ICQRIT/SRECOM in 2003, and the Best Paper Award from the Society of Project Management in 2006. He is a regular member of the IEEE, IPSJ, ORSJ, JSQC, SPM, the Japan SIAM, REAJ, JIMA.
