QUALITY AND RELIABILITY ENGINEERING INTERNATIONAL Qual. Reliab. Engng. Int. 2004; 20:61–79 (DOI: 10.1002/qre.547)
Research
Response Modeling Methodology Validating Evidence from Engineering and the Sciences Haim Shore∗,† Department of Industrial Engineering, Faculty of Engineering Science, Ben-Gurion University of the Negev, P.O. Box 653, Beer-Sheva 84105, Israel
Modeling a response in terms of the factors that affect it is often required in quality applications. While the normal scenario is commonly assumed in such modeling efforts, leading to the application of linear regression analysis, there are cases when the assumptions underlying this scenario are not valid and alternative approaches need to be pursued, like the normalization of the data or generalized linear modeling. Recently, a new response modeling methodology (RMM) has been introduced, which seems to be a natural generalization of various current scientific and engineering mainstream models, where a monotone convex (concave) relationship between the response and the affecting factor (or a linear combination of factors) may be assumed. The purpose of this paper is to provide the quality practitioner with a survey of these models and demonstrate how they can be derived as special cases of the new RMM. A major implication of this survey is that RMM can be considered a valid approach for quality engineering modeling and, thus, may be conveniently applied where theory-based models are not available or the goodness-of-fit of current empirically-derived models is unsatisfactory. A numerical example demonstrates the application of the new RMM to software reliability-growth modeling. The behavior of the new model when the systematic variation vanishes (there is only random c 2003 John Wiley & Sons, Ltd. variation) is also briefly explored. Copyright KEY WORDS: generalized linear modeling; quality improvement; transformations; variation modeling
1. INTRODUCTION
E
fforts to improve quality often require that the factors affecting a quality response will be identified and the relationship between the factors and the investigated response be modeled. The output of this endeavor allows either the identification of root-causes that account for low quality or the determination of optimal settings in the framework of robust parameter design. When the common assumptions underlying the normal scenario (that is, additivity of effects, constant error-variance and normality of errors) are valid, linear regression is the preferred option for data analysis and modeling. Conversely, when the assumptions of
∗ Correspondence to: Haim Shore, Department of Industrial Engineering, Faculty of Engineering Science, Ben-Gurion University of the
Negev, P.O. Box 653, Beer-Sheva 84105, Israel. † E-mail:
[email protected]
Published online 18 December 2003 c 2003 John Wiley & Sons, Ltd. Copyright
Received 27 June 2001 Revised 22 September 2002
62
H. SHORE
the normal scenario are not met, alternative approaches need to be pursued, like the ‘normalization’ of data (via some normalizing transformation) or generalized linear modeling (GLM). Recently, a new response modeling methodology (RMM) has been developed for modeling a response in a normal/non-normal environment1,2 . The only restrictive assumption of the new methodology is that the relationship between the response and the affecting factor (or a linear combination of factors) is monotone convex (concave). The purpose of this paper is to demonstrate that the new RMM is in fact a natural generalization of existing models, developed over the years in a wide spectrum of engineering and scientific disciplines. By surveying some of these models and showing how these can be derived as special cases of the new RMM, we hope to provide some validating and supportive evidence for the general use of the new modeling approach in quality and reliability engineering applications. Furthermore, we numerically demonstrate the effectiveness of RMM relative to existing models. To motivate exposition of the new RMM, it is perhaps useful to first provide a brief review of the current general approaches to response modeling and consider how they address ‘data incompatibility’ with the normal scenario. The ‘data normalization’ approach attempts to achieve the conditions of the normal scenario by a single transformation of the available data. A common practice is to use the Box–Cox transformation for continuous variables and various dedicated transformations for attributes data. Examples of the latter are the arcsin transformation for binomial data or the square root transformation for Poisson data (a recent review of these transformations and others may be found in Shore3 ). Alternatively, we may opt for the use of GLM. This has recently gained favor with quality practitioners and, at times, advocated as having a better performance relative to the normalizing approach (refer to Lewis et al.4 ). Unlike the normalizing approach, GLM does not require normality of errors; however, within the modeling process two separate decisions need to be made. The first decision requires specification of the distribution of the error (or, alternatively, specifying the variance-function, which expresses the variance in terms of the mean). The second decision regards the scale needed to achieve additivity of the systematic effects (transforming of the mean, via a link function, to obtain a ‘linear predictor’). Thus, while the normalization approach attempts to achieve all three conditions of the normal scenario via a single data transformation, GLM models the given response by separating the modeling of the error from the modeling of the mean. Good recent introductions of the GLM approach related to quality improvement efforts and some numerical comparisons between the two approaches may be found in an introductory tutorial on GLM5,6 and also in a recent book on the subject7 . While the above general systems for response modeling in a non-normal environment are common within the quality discipline, modeling the relationship between a response and the factors that may explain its systematic variation is not confined to this area. Indeed, modeling efforts that attempt to explain the variation of a response in terms of the variation in some relevant factors comprise the bulk of the scientific and engineering research effort. Preliminary surveys2,8 of some of these models reveal that in many cases the studied (unknown) relationship between the response and the affecting factor (or a linear combination of factors) is either convex or concave and that it is uniformly so throughout the interval of interest. Based on this observation, the new RMM has been developed and shown to be a natural generalization of these models. Thus, unlike the ‘normalization’ approach and GLM, which originated in models of random variation (the normal distribution and the exponential family of distributions, respectively), the new RMM has its roots in current scientific and engineering models, where the relationship between the response and the ‘linear predictor’ may be assumed to be monotone convex (or concave). In this paper, we expand on the earlier preliminary surveys and demonstrate, for a broader spectrum of disciplines and models, that the new RMM indeed provides a natural ‘umbrella’ for many of these models. Furthermore, we will show that a class of inverse normalizing transformations, recently developed to provide a general representation for non-normal distributions9,10 and later applied to derive general control charts for variables11, are special cases of the new model. A general implication for the quality discipline of the survey provided here is that the new methodology may be applied to a wide spectrum of scenarios, where a monotone convex/concave relationship is discerned, virtually ‘anxiety-free’. A quality practitioner is often required to model a quality response when theory-based models are non-existent, or when current theory-based or empirically-derived models do not provide satisfactory goodness-of-fit. Showing that the new response-modeling approach provides a natural generalization of models, c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
63
derived over the years in a myriad of engineering and scientific disciplines, would hopefully provide the quality practitioner with sufficient validating evidence so that the new approach may be applied with a high degree of assurance that good results (high goodness-of-fit) can be anticipated. The structure of the rest of the paper is as follows. We start by introducing (in Section 2) three preliminary examples of existing models and address some features which they all share in common. Pursuing the original presentation of the new methodology in Shore2, we provide in Section 3 the expressions needed to apply the new approach and outline a maximum likelihood procedure. For the interested theory-oriented reader, a brief description of the axiomatic derivation of the model is provided in Appendix A. Section 4 expands on the preliminary examples of Section 2 and mainstream models in four engineering and scientific areas are addressed: Chemistry and chemical engineering (Section 4.1), physics (Section 4.2), electrical engineering (Section 4.3) and nonlinear growth models (Section 4.4). We show how these models can be derived from the new RMM. In Section 5, we apply the new approach to software reliability-growth data. It is demonstrated that the residuals standard deviation obtained by the new modeling approach is appreciably lower than that attained with any of the current mainstream models. In Section 6, we explore the structure of the model when the systematic variation vanishes and the response experiences only random variation. Section 7 discusses the results and their implications.
2. THREE PRELIMINARY EXAMPLES In this section, we address three existing models for a response that is related to temperature. The purpose of these preliminary examples is to intuitively motivate the error structure presumed by the new RMM and to demonstrate the monotone convex/concave relationship that all models derived from RMM share. In Section 4, these models will be re-addressed by showing how they may be easily derived in the framework of RMM. The first two examples are from chemistry and chemical engineering. They are taken from Daubert13, where various equations for correlating thermodynamic and transport properties with temperature are surveyed and compared. The third example is from electric engineering, based on Halliday et al.14. Both of these sources also serve to outline the models in Section 4 (wherein we also elaborate on why these references were selected as the source for the models of this survey). Example 1 (chemistry) This example relates to the modeling of vapor pressure, P , in terms of temperature, T (on a Kelvin scale). Perhaps the most well-known and widely used is the Antoine equation15 which is over a century old and is derived from theoretical principles16 that are still valid today: log(P ) = A + B/(T + C)
(1)
Riedel17 modified (1) by adding two terms, to obtain log(P ) = A + B/T + (C) log(T ) + DT E
(2)
Equation (2) is judged by Daubert13 to be the most adequate for modeling temperature dependence of vapor pressure today, however this is basically the Antoine equation improved empirically by adding two polynomial terms. Let us address the Antoine equation as given in (1). This expression does not provide an indication as to the structure of the error. Let us assume first that temperature is constant (no systematic variation or random error is associated with temperature). Obviously, the measured pressure will still fluctuate randomly and this random variation will be unrelated to temperature (since the latter is assumed constant). Natural instability of pressure caused by factors other than temperature (for example, fluctuations in vapor density), as well as measurement imprecision, may contribute to this random variation. Thus, we may re-write (1), with the pressure-related random error, ε2 : log(P ) = A + B/(T + C) + ε2 c 2003 John Wiley & Sons, Ltd. Copyright
(3) Qual. Reliab. Engng. Int. 2004; 20:61–79
64
H. SHORE
Other forms of random error may be conceived relative to P (assuming that T is still constant), for example log(P + ε2 ) (see Appendix A). Now discard the assumption that T is constant and assume that T itself is associated with a random error. It is natural to assume that random deviations in temperature will be transmitted to the measured pressure or, in other words, that a second source of random variation that affects pressure may naturally be assumed. Denote the random deviation associated with temperature by ε1 . Re-writing T = µT + ε1 , where µT is the expected value of the measured T and ε1 is assumed to be independent of ε2 , we finally obtain the Antoine equation with the errors log(P ) = A + B/(µT + ε1 + C) + ε2
(4)
Observing this expression, it is obvious that the relationship between the response (P ) and the affecting factor (T ) is monotone (always increasing) and that it may be characterized as an exponential–power relationship. Example 2 (chemical engineering) For surface tension, S, Daubert13 recommends the use of Guggenheim’s empirical equation18, written here with the errors S = A(1 − T /Tc )C + ε2 = A[1 − (µT + ε1 )/Tc ]C + ε2
(5)
and Tc is the critical temperature (a constant). Note, that this time the additive error, ε2 , relates to the original response and not to the log transformed response. From (5), the relationship between the response, S, and T may be characterized as a power relationship. Example 3 (electric engineering) A thermistor is a semiconductor device with a temperature-dependent electrical resistance. It is used in medical thermometers and to sense over-heating in electronic equipment. Over a limited range of temperature (T > Ta , Ta given), the resistance, R, is (see Halliday et al.14, p. 547) R = Ra exp[B(1/T − 1/Ta )],
(6)
T > Ta
where B is a constant that depends on the particular semiconductor used and Ra is the resistance at T = Ta . Re-written with the errors, (6) becomes R = Ra exp{B[1/(µT + ε1 ) − 1/Ta ] + ε2 },
T = µT + ε 1 > T a
(7)
From (7), the relationship between the response, R, and T may be characterized as an exponential–power relationship. From these three preliminary examples and others (refer to Section 4), two observations emerge that characterize all models. Observation A Variation in the response may be attributed to two independent sources of variation: ‘self-generated variation’ and ‘externally-generated variation’ (the motivation for selecting the given names to denote the two sources of variation will be explained later on). While the former consists of only random variation, the latter comprises both a systematic component of variation and a random component of variation in the form of an additive random error. c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
65
Observation B Certain patterns, which are supposed to capture uniformly convex relationships between a response and the affecting factor repeatedly appear in models developed independently in various scientific and engineering disciplines. Two examples were provided in the above models: power and exponential–power relationships. However, other relationships, like linear, exponential or exponential–exponential also appear often (as the survey of this paper will demonstrate). In fact, these various basic functions may be arranged in a hierarchical order, which reflects a growing ‘intensity’ of convexity. This hierarchy is captured in the ‘ladder of fundamental uniformly convex functions’, to be addressed in more detail in Appendix A. In the pursuing section, we show how the two empirical observations, made with regard to the above examples, naturally emanate from the new RMM.
3. THE GENERAL MODEL Presentation of the general model in this section pursues Shore1,2 . For the convenience of the more theoreticallyoriented reader, the technical details of the axiomatic derivation of the model are deferred to Appendix A, and only the main expressions associated with the new RMM are given here. In developing the general model, we attempt to capture the behavior of a random response, which experiences two independent sources of variation (hopefully already familiar to the reader from the previous examples): self-generated variation and externally-generated variation. The latter variation is assumed to comprise both systematic variation (transmitted to the response by changes in the mean of the affecting factor) and random variation (transmitted to the response by an additive normal error, ε1 ). When several external factors affect the response, the carrying agent is denoted ‘the linear predictor’, η similar to the corresponding term used in GLM. The externally generated variation is modeled by f1 (η, Z1 ; θ1 ), where θ1 is a vector of parameters and Z1 is a standard normal variable, which generates the random error ε1 = σε1 Z1 (namely, the error is normally distributed with zero mean and standard deviation σε1 ). It is assumed that f1 (η, Z1 ; θ1 ) is monotone convex or concave (with respect to η). In the rest of the paper we only relate to a convex relationship (f1 is uniformly bending upward); however, it should be understood that any assertion with respect to a monotone convex relationship also applies to a monotone concave relationship. Self-generated variation reflects the inner natural instability of the modeled response, as well as measurement imprecision. This variation is modeled by f2 (Z2 ; θ2 ), where θ2 is a vector of parameters and Z2 is a standard normal variable, assumed to be independent of Z1 . A basic assumption of the model is that self-generated variation is proportional to the magnitude of the response. The general model expresses the response, Y , in terms of the linear predictor, η, the random variables, Z1 and Z2 , and the model’s parameters: Y = f1 (η, Z1 ; θ1 )f2 (Z2 ; θ2 ) = exp{(α/λ)[(η + σε1 Z1 )λ − 1] + µ2 + σε2 Z2 }
(8)
where ε1 = σε1 Z1 and ε2 = σε2 Z2 represent independent random errors and θ1 = {α, λ, σε1 }, θ2 = {µ2 , σε2 } are parameters’ vectors associated with f1 and f2 , respectively (details for the derivation of this model can be found in Appendix A). The reader may easily verify that the models of Section 2 (and those in the afore-cited references) are either exact special cases of the general model (Equation (8)), or slightly modified versions of it (linear transformations at most). Next let us develop expressions for the response mean and for the response variance. Since Z1 and Z2 are assumed to be independent, so are f1 (η, Z1 ; θ1 ) and f2 (Z2 ; θ2 ) and we may write the mean and variance of Y in Equation (8) and the mean of log(Y ) as E(Y ) = E[f1 (η, Z1 ; θ1 )]E[f2 (Z2 ; θ2 )] ∼ = exp[(α/λ)(ηλ − 1) + µ2 + (σ 2 /2)]{1 + (σ 2 /2)[η2(λ−1)α 2 + ηλ−2 α(λ − 1)]} ε2
c 2003 John Wiley & Sons, Ltd. Copyright
ε1
Qual. Reliab. Engng. Int. 2004; 20:61–79
66
H. SHORE
V (Y ) = V [f1 (η, Z1 ; θ1 )]E{[f2 (Z2 ; θ2 )]2 } + V [f2 (Z2 )]E{[f1 (η; Z1 )]}2 E[log(Y )] = E{(α/λ)[(η + σε1 Z1 )λ − 1]} + µ2 ∼ = (α/λ)[(η)λ − 1] + (1/2)α(λ − 1)σ 2 ηλ−2 + µ2
(9)
ε1
where the approximate expressions for E(Y ) and E[log(Y )] have been derived by expanding f1 , in terms of Z1 , into a Taylor series around zero and taking the expectation of the first four terms. Note that since both f1 and f2 are exponential functions, once the means of each are known, there is little difficulty in finding the means of (f1 )2 and (f2 )2 , required to calculate the variance. Thus, the kth non-central moment (moment about zero) of f1 is obtained by replacing α in the expression for the mean by (kα), namely (refer to Equations (8) and (9)) 2 E{[f1 (η, Z1 ; θ1 )]k } = exp[(kα/λ)(ηλ − 1)]{1 + (σε1 /2)[η2(λ−1)(kα)2 + ηλ−2 (kα)(λ − 1)]}
(10)
Examining the two expressions for the mean (Equation (9)), we realize that both are very versatile in their ability to model variously-shaped link functions (a link function is a transformation of the response mean needed to obtain the linear predictor). In particular, they provide the link functions common within the GLM framework. 2 in Equation (9) (assume σ 2 = 0), we obtain for λ = 0 and α = 1 Thus, if we ignore the term that contains σε1 ε1 a linear link function, while λ = 0 and α = −1 result in a reciprocal relationship between the linear predictor and the mean, typical for the gamma (exponential) error distribution within the GLM framework. A value of λ = 1 would result in a log link function, typical to the Poisson distribution. We realize that the values of the parameters, in particular that of λ, are critical in determining the final structure of the model. Since λ is estimated within the data-analysis procedure (refer to the outline of the estimation procedure provided below), no specification of the model structure is needed prior to data analysis. A detailed description of an approximate maximum likelihood estimation procedure associated with the new model is given in Shore1,2 and the reader is referred to these sources for the theoretical development and some numerical examples. We will now provide a brief outline of this procedure. The approximate maximum likelihood procedure assumes that σε2 = 0 and then proceeds in an iterative mode, where it alternates between weighted nonlinear regression, applied to LT = log(Y ) to find estimates of the parameters {α, λ, µ2 }, and linear regression, applied to appropriately transformed data to find estimates of the coefficients of the linear predictor and of σε1 . The weights for the nonlinear regression are the reciprocal values of the error variances, which are estimated for individual observations. These estimates may be derived either from available data (if replicated observations are available), or by numerical integration of LT , based on the log of Equation (8). This iterative procedure terminates when no further improvement is noticed in the fit obtained from the nonlinear regression. The approximate estimation procedure will be demonstrated in the numerical example of Section 5.
4. SURVEY OF CURRENT MAINSTREAM MODELS AND THEIR DERIVATION FROM RMM In this section, we survey some well-known and commonly applied mainstream scientific and engineering models. As alluded to earlier, the focus in this paper is on presenting existing mainstream models and demonstrating how we may derive these as special cases of the new model. Four different subject areas are included in this survey: chemistry and chemical engineering, physics, electric engineering and nonlinear growth models. All the current models will be introduced in a uniform format: definition of the response; the affecting factors and the model’s parameters, and specification of the model (including reference to the source). No theoretical background for the derivation of the models is provided as this may be easily retrieved from the quoted dedicated references. Three main sources for this survey are: Daubert13 for chemistry and chemical engineering; Halliday et al.14 for physics and electric engineering (two further sources for particular models are also used); and Myers et al.7 for growth models. Our main consideration in selecting these sources is that they provide mainstream models that have been widely recognized as such by the respective community of scientists or engineers. Thus, Daubert13 surveyed models developed over the years for various chemical responses that are affected by temperature and based on currently available data-sets, has formulated recommendations as to which models c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
67
best suit the data. Accordingly, only those models recommended by Daubert will be alluded to. Similarly, Halliday et al.14 is a basic textbook in physics (and related engineering applications) for undergraduate college students and as such is expected to present mainstream models. Myers et al.7 present growth models used in biology, economics and engineering. We use the nomenclature and notation employed in these sources and the reader is encouraged to become familiar with them in order to appreciate the value of the uniform approach for response modeling introduced by the new methodology. Finally, we note that we are aware that the models selected for this survey do not necessarily constitute a representative sample of mainstream models and other models could have been selected. However, we consider this as of secondary importance since this survey intends to serve only as a demonstration for the universal nature of the new model and its capability to represent and capture the common features shared by current mainstream models. 4.1. Chemistry and chemical engineering In this section, we examine several empirical or theory-based relationships that have appeared in the literature. We start with the temperature dependence of vapor pressure. Referring to Equation (4), we easily recognize that this equation is derivable from (8), with parameters η = µT + C, α = −B; λ = −1, µ2 = A + B. Equation (2) is a polynomial, and will be addressed separately. Correlation of heat of vaporization with temperature must be consistent with the exact Clapeyron equation dP/dT = Hv /(TV), where Hv is the difference in the enthalpies of saturated vapor and saturated liquid. Theoretical considerations require that Hv will be modeled in terms of t = 1 − Tr (where Tr = T /Tc , is the temperature relative to its value at the critical point, Tc ). Dauber13 introduces three current models: Hv = A[B d − 1] Hv = At B
(11)
Hv = Bt A + Ct (2A) + Dt (3A) + · · · where d = t C . Regarding these equations, we realize that the two upper equations are special cases of (8), while the third equation is again a polynomial which will be addressed separately. For solid density, Daubert13 notes that ‘as the variation with temperature is not strong, a linear equation is normally adequate with a data range of the lowest data point to the triple point of the compound. In a few cases with a large amount of data, a simple quadratic polynomial was recommended. For most situations use of a single exponential value over the entire temperature range is sufficient’. All these cases are simple special cases of (8). For liquid density, ρL , Racket19 suggested an equation which Daubert13 concludes ‘is recommended for correlation of liquid density’. Written here with the errors, we obtain for Racket’s equation log(ρL ) = A + B[1 − (µT + ε1 )/C]D + ε2
(12)
where A, B, C and D are parameters that need to be determined. It is easy to see that this equation is derivable from (8) with: η = 1 − µT /C, α = BD, λ = D, µ2 = A + B. For solid heat capacity, Daubert13 recommends a simple polynomial in temperature, where ‘most data can be fit with a linear equation, with a quadratic necessary for a few systems’. It is simple to realize that both linear and quadratic expressions are special cases of (8). For liquid heat capacity, CPL , Daubert13 recommends as a default the use of the following equation, originally suggested by Zabransky et al.20 : CPL = A2 /t + B − 2(A)(C)t − (A)(D)t 2 − (C 3 /3)t 3 − (C/2)(D)t 4 − (D 2 /5)t 5
(13)
with t = 1 − Tr . This is a polynomial in t, which we will address shortly. For ideal gas heat capacity, Daubert13 notes that most suggested relationships are of polynomial form. However, he recommends the use of the following exponential equation (Equation (14-1) therein): CP0 = A + B exp(−C/T D ) c 2003 John Wiley & Sons, Ltd. Copyright
(14) Qual. Reliab. Engng. Int. 2004; 20:61–79
68
H. SHORE
Re-writing Equation (14) with the random errors, we obtain CP0 = A + B exp[−C/(µT + ε1 )D ] + ε2
(15)
It is easy to see that this equation is a linear transformation of (8), derived thereof with: η = µT ; α = (−D)(−C); λ = −D; µ2 = log(B) − C. Note, that the additive error, ε2 , as given in (15), is another permissible mode of specifying the random error associated with self-generated variation and is different from that given in Equation (8) (refer to Appendix A for details). For temperature dependency of liquid viscosity, µL , Daubert13 recommends that the following equation be used for correlation purposes: log(µL ) = A + B/T + C log(T ) + DT E
(16)
This is again in a polynomial form, to be addressed shortly. For thermal conductivity, Daubert13 examines both liquid and low-pressure vapor thermal conductivity. All of the examined relationships are either polynomials in T or in t = 1 − Tr = 1 − T /Tc . For surface tension, S (refer to Section 2), it is easy to verify that Equation (5) is derivable from (8) with parameters: η = 1 − µT /Tc ; α = C; λ = 0; µ2 = log(A). We now refer to the polynomial models. Polynomials are a linear combination of various powers of the ‘linear predictor’ and therefore constitute a linear combination of various terms derived from Equation (8) with the special value λ = 0. Since an implicit objective of using polynomials is often to capture various degrees of ‘convexity’, we may reasonably expect Equation (8) to provide a good substitute for polynomials due to its versatility in representing convexity. A demonstration for this is given by viscosity modeling, delivered by the next two expressions in Daubert13 (Equations (21) and (22) respectively, therein): µL = exp[A + B/T + C log(T ) + DT E ] µL = exp(AF ) − C
(17) (18)
where F = T B , and Daubert’s equations are presented here in somewhat different forms. Daubert states that (17) and (18) are nearly equivalent in the degree of accuracy they deliver. Yet (17) is a polynomial, while (18) is obviously a special case of (8) (up to a linear transformation). Our own experience with various existing cases of modeling by polynomials confirms that Equation (8) regularly delivers a good substitute for a polynomial representation when the modeling of uniformly convex (concave) relationships is required. Since increasing the number of parameters (afforded by the use of a polynomial) may enhance accuracy but also increase the risk of unstable parameters’ estimates (for an example refer to Shore et al.8 ), the issue of whether a polynomial should be used has to be carefully balanced against the alternative, offered by the new approach. 4.2. Physics In this section, we address some well-known physical relationships derived either from Newtonian principles, current cosmological theories or quantum mechanics. We pursue here three sources: Halliday et al.14, a basic textbook of physics for undergraduate college students, Benedek and Villars21 and Schroeder22. To simplify presentation of the analogy with Equation (8), only the values of λ and α, which determine the model structure, will be specified. While attempting to illustrate the trait of self-similarity typical to power laws, Schroeder22 (see p. 33 therein) emphasizes that ‘homogenous power laws, like Newton’s universal law of gravitational attraction, abound in nature, dead and alive alike’. We begin therefore with some examples of power laws in Newtonian physics. A well-known basic relationship, which binds kinetic energy (Ek ) with velocity (V ) is Ek = M(V )2 /2
(19)
where M is a proportionality coefficient (the body mass). Similar quadratic relationships hold for an object moving in a circle (between centripetal force, Fc , and velocity, V ), for a spring (between the elastic potential c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
69
energy and the spring’s elongation (compression) from its rest position, X), and for a body that falls through air (between the drag force, D, and the object’s velocity, V ). Quadratic expressions abound in various areas of physics. However, there are also other power relationships with non-quadratic exponents. For example, the radius of the earth, R, as a function of its mass, M, given its mean density, ρ, is R = {[3/(4π)](M/ρ)}(1/3)
(20)
It is easy to realize that all these power relationships are special cases of (8). Suppose that the temperature differential between an object and its surrounding is T . Then, according to Newton’s law of cooling, if at some instant t = 0, the temperature difference is T0 , then at some time later, t (> 0), the temperature difference will be (see Halliday14, p. 548 with the errors added) T = T0 exp[−a(t + ε1 ) + ε2 ]
(21)
where a is a constant. This model may be derived from (8) with the parameters λ = 1, α = −a. Every solid (and liquid) at finite temperature is in equilibrium with its own vapor. The saturation vapor pressure, Pvapor , depends on the temperature: T = µT + ε1 , mainly through the Boltzmann factor (see Schroeder22, pp. 436 and 442 with the errors added): Pvapor ∼ exp{−EB /[k(µT + ε1 )] + ε2 }
(22)
where k is the Boltzmann factor, T is on a Kelvin scale and EB is a particle binding energy (find details therein). This model may be derived from (8) with the parameters λ = −1 , α = EB /k. The ‘Barometric formula’ is a well-known expression for the exponential decrease of the density of a gas (at constant temperature) with height, assuming a ‘constant’ temperature T . Let M be the molecular weight, R be the gas constant and g be the gravitational constant. Then, at height z the spatial density, ρ, depends on the ratio K = z/T via the relationship (see Schroeder22, p. 429 with the errors added) ρ(K) = ρ(0) exp{−(Mg/R)(K + ε1 ) + ε2 }
(23)
This model may be derived from (8) with the parameters: λ = 1, α = −(Mg/R). Next we move to some relationships from relativity and quantum physics. Rest energy plus kinetic energy for an object with rest mass of m, moving at a speed v, near the speed of light c (from Einstein’s special theory of relativity) E(L) = mc2 (1 − L)−1/2
(24)
where L is the relative speed squared, L = (v/c)2 . This power relationship may be derived from (8) with the parameters η = 1 − L, λ = 0, α = − 12 . Planck’s radiation law is (see Halliday et al.14 , p. 1140 with the errors added) S(T0 ) = (2πc2 h/λ50 ){exp[(hc/λ0 k)/(µT0 + ε1 )] − 1}−1
(25)
where S is spectral radiancy, λ0 is the wavelength at temperature T0 = µT0 + ε1 , and h is the Planck constant. The response (2πc2 h/λ50 )/S(T0 ) + 1 (a linear transformation of 1/S(T0 )) may be derived from (8) with the parameters λ = −1, α = −(hc/λ0 k). The energy levels of the stationary states of the hydrogen atom are given by (see Halliday et al.14 , p. 1147) E(n) = −13.6 eV/n2 ,
n = 1, 2, 3, . . .
(26)
where n is the quantum number and the numerical coefficient is the electron ground-state energy, measured in electron-volts (eV). This model may be derived from (8) with the parameters η = n, λ = 0, α = −2. The variable n is an integer and since this is a theoretical model, no errors are added. c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
70
H. SHORE
The radioactive decay rate, R, is given by (see Halliday et al.14, p. 1239 with the errors added) R(t) = (R0 ) exp[−k(t + ε1 )] + ε2
(27)
where at time t = 0 the rate is R0 . This model is derivable from (8) with the parameters λ = 1, α = −k. Note that the error, ε2 , is assumed to comply with the error structure specified by (A2a) (see Appendix A for details). The neutron generation time, tgen , in a nuclear reactor is the average time needed for a fast neutron emitted in one fission to be slowed down to thermal energies by the moderator and to initiate another fission. Let P0 be the power output of the reactor at time t = 0. Then, the output at time t, P (t), is given by (see Halliday et al.14 , p. 1279 with the errors added) P (t) = P0 exp[(k/tgen )(t + ε1 ) + ε2 ]
(28)
where k is a constant. This model may be derived from (8) with the parameters λ = 1, α = k/tgen . 4.3. Electric engineering Coulomb’s law specifies the electrostatic force of attraction (or repulsion), F , between two point charges, q1 and q2 , separated by a distance r. It is given by (with the errors added) F (r) = k(q1 q2 )/(r + ε1 )2 + ε2
(29)
where k is a constant. This model is a power law, derivable from (8) with the parameters λ = 0, α = −2. A capacitor of capacity C, which is initially uncharged (q = 0 at t = 0), is charged by an ideal battery with an emf (electromotive force) E. At time t, the voltage across the capacitor, as measured by a resistor connected in parallel, with resistance R, is (see Halliday et al.14, p. 805 with the errors added) VR (t) = (E) exp[−(t + ε1 )/RC + ε2 ]
(30)
Similarly, a discharging capacitor, with an initial (full) charge of q0 , will have at time t a charge of (with the errors added) q(t) = q0 exp[−(t + ε1 )/RC + ε2 ]
(31)
Similar equations apply for the rise (or fall) of the current produced by an emf E which is introduced into (or removed from) a single loop circuit containing a resistor R and an inductor L (see Halliday et al.14, p. 903). Both (30) and (31) are easily recognized as special cases of (8). Suppose that an electrical current of intensity i runs through a loop of radius R (area S). The magnetic field, B(z), produced at a point located a distance z along the loop axis is parallel to the axis and is given by (see Halliday et al.14 , p. 863 with the errors added) B(z) = (µ0 /2π)(iS)/(z + ε1 )3 + ε2 = (µ0 /2)(iR 2 )/(z + ε1 )3 + ε2
(32)
where µ0 is a permeability constant. This model may be derived from (8) with the parameters λ = 0, α = −3. 4.4. Growth models Myers et al.7 introduce growth models as a special category of nonlinear regression models. Here we pursue their presentation of some mainstream models (therein, Section 3.6). We use growth models to describe how a response grows with changes in a regressor variable. Typical applications are in biology, where plants and organisms grow with time, but there are also many applications in economics and engineering. Some such models, used in software reliability-growth models, are presented in Shore2 and also briefly outlined in the numerical example of Section 5. c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
71
The logistic growth model is y = β1 /[1 + β2 exp(−β3 x)] + ε
(33)
y = β1 exp[−β2 exp(−β3 x)] + ε
(34)
y = β1 − β2 exp(−β3 x β4 ) + ε
(35)
The Gompertz model is
The Weibull model is
where {β1 , β2 , β3 , β4 } are parameters that need to be determined. Apparently, the first model is a power– exponential model, the second an exponential–exponential model and the last is an exponential–power model, all of which are easily recognized as special cases of (8).
5. A NUMERICAL EXAMPLE In an earlier paper, Shore2 has demonstrated that most currently used software reliability-growth models are in fact special cases of (8) or linear transformations thereof. In this section, we use some previously published data to demonstrate that better goodness-of-fit is achieved with the new model than with any of the existing mainstream software reliability-growth models. We start with a brief exposition of five current mainstream models, and then proceed to analyze the data that serve in this example. Software reliability-growth models attempt to model the number (or rate) of software failures as a function of time. In many cases, it is assumed that as more errors are detected and corrected and no new errors are injected into the software as a result of the repair activities, the reliability of the software will tend to grow. This assumption forms the basis for a widely-used category of models generally referred to as reliability-growth models. The most common applications in this category are variations of a basic model, in which the number of failures that have occurred by time t is modeled by a non-homogenous Poisson process (NHPP), with mean value function µ(t). In other words, the probability function for the number of failures, Ni , in the time interval [ti−1 , ti ), is given by Pr(Ni = n) = f [n|µ(ti ) − µ(ti−1 )] = {[µ(ti ) − µ(ti−1 )]n /n!} exp{−[µ(ti ) − µ(ti−1 )]}
(36)
that is, a Poisson probability function with time-dependent parameter µ(ti ) − µ(ti−1 ). NHPP models differ by how they model the mean function and these are divided into two main categories: finite failure models, where it is assumed that µ(∞) is finite; and infinite failure models, where the mean value is not bounded for large values of t. Examples for the former category are as follows. I. Goel and Okumoto23, who have assumed an exponential mean function µ(t) = a[1 − exp(bt)],
a > 0,
b>0
(37)
II. the delayed S-shaped model (Yamada and Osaki24 ) µ(t) = a[1 − (1 + bt) exp(bt)],
a > 0,
b>0
(38)
Examples for the latter category are as follows. III. Musa et al.25 , who have proposed the logarithmic Poisson model µ(t) = (a) log(1 + bt) c 2003 John Wiley & Sons, Ltd. Copyright
(39) Qual. Reliab. Engng. Int. 2004; 20:61–79
72
H. SHORE
Table I. Results from the analysis of Musa’s30 data set M3 Model
a
b
LL∗
STDres
I II III IV V
189 638 160 952 338 825 0.9342 0.02345
0.009 551 0.032 01 0.005 169 3.0667 11.45
−69 365 449 633 432 702 470 961 468 773
10 464 8147 10 154 2988 2962
IV. The Duane model26, which assumes a power-law process (PLP) µ(t) = αt β ,
α > 0,
β >0
(40)
V. The log–power model introduced by Xie and Zhao27 µ(t) = (a)[log(1 + t)]b ,
a > 0,
b>0
(41)
VI. The generalized power family models (GPFMs) by Knafl and Morgan28 µ(t) = α[k(t)],
α > 0,
β >0
(42)
where k is a positive and strictly increasing known function of t, with k(0) = 0. Arnoux et al.29 suggest criteria for choosing appropriate GPFM model, using Musa’s data30 . A good review of these models and others may be found in Xie31 , which also served as a major reference for the above short review. In this section, we introduce a detailed numerical example for analyzing software reliability-growth data, using the new general model. We will compare the results of this analysis to results obtained from models I–V, with parameters that have been estimated by a maximum likelihood procedure. The input for the analysis of this example is Musa’s M3 data set (sample size of k = 38). Musa’s data sets30 have become industry-standard when the appropriateness of various software reliability-growth models is examined and we pursue previous references in that regard (for example, Arnoux et al.29 ). We start by deriving maximum likelihood estimates for models I–V above. Given a data set of k observations,{n1, n2 , . . . , nk }, where ni is the number of failures during time interval [ti−1 , ti ), the log likelihood function, assuming an underlying Poisson distribution, is LL = log
k
f [ni |µ(ti ) − µ(ti−1 )] = −
i=1
= −µ(tk ) +
k
[µ(ti ) − µ(ti−1 )] +
i=1 k
ni log[µ(ti ) − µ(ti−1 )] −
i=1
k i=1
k
log(ni !)
ni log[µ(ti ) − µ(ti−1 )] −
k
log(ni !)
i=1
(43)
i=1
For a given model for the response mean, the parameters’ values that maximize (43) will provide maximum likelihood estimates. Identifying maximum likelihood estimates for models I–V, these estimates, together with values of LL at the optimal point (LL∗ ) and the associated standard deviations of the residuals (STDres), are given in Table I. It is evident, both from the values of LL∗ and those of STDres, that only models IV and V provide acceptable accuracy (goodness-of-fit). Scatter-plots of the residuals from these models are given in Figure 1. Referring next to the new approach, no specific mean function is assumed and the main determinant of the final structure of the mean function is the value of λ, as derived from the data analysis. However, the form of the linear predictor, as a function of t, has to be decided. We start with the simplest model and assume η(t) = t c 2003 John Wiley & Sons, Ltd. Copyright
(44) Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
73
Note, that from a theoretical point of view this is the preferred choice since this η embodies the ideal of the new approach, namely simplicity in the definition of the linear predictor. Only if this choice does not provide results competitive with those obtained by the current models (models I–V) will we opt for an expanded linear predictor, which will also include a coefficient and a constant. Thus, we obtain the response quantile from (8) y = exp{(α/λ){[(η + σε1 z1 )λ − 1]} + µ2 + σε2 z2 } = exp{(α/λ){[t + σε1 z1 ]λ − 1} + µ2 + σε2 z2 }
(45)
where y is the cumulative number of failures by time t. From (45), neglecting (σε2 )z2 , we obtain for the ith observation the transformed value ti + σ ε1 z1i = [(λ/α)[log(yi ) − µ2 ] + 1](1/λ)
(46)
where (σ ε1 )z1i is the (unknown) random error associated with the ith value of the linear predictor, ti . The data transformation given by (46) serves as an input for the linear regression part of the analysis. Only if a linear transformation of t, namely η = a0 + a1 t, delivers a much improved fit in the linear regression analysis (namely, {a0 , a1 } are appreciably different from (0,1), respectively), will we change the selected linear predictor (Equation (44)). We first derive initial estimates of the parameters {α, λ, µ2 }, using LT = log(Y ) as the data input in a nonlinear regression procedure and E(LT ) as given by (9). However, for this first iteration we assume that σε1 = 0 in the expression for E(LT ). In subsequent iterations, an estimate of σ ε1 will be available from the most recent application of the linear regression stage of the analysis and this estimate may be used in the expression for E(LT ). Also, no weighting is introduced in the first iteration. In subsequent iterations, we will use weighting only if the residual variance differs appreciably from one observation to the next, namely σ ε1 is not negligible relative to typical values of η (refer to Equation (45)). A first run using the approximate procedure delivers: α = 0.7433, λ = 0.4809, µ2 = 3.8113, with STDres = 1817. The latter statistic is about 61% of the best of the STDres in Table I. Observing the residuals we find observation 31 to have an outlying residual of 6256.6. Excluding this observation from the analysis we obtain α = 0.4335, λ = 0.6437, µ2 = 4.832 with STDres = 1471, which is about 50% of the best of the STDres which appear in Table I. Turning now to the linear section part of the analysis, we model the linear predictor as a0 + a1 t, with the above parameters of {α, λ, µ2 } (refer to Equation (46)). We obtain a0 = 1.5211, a1 = 0.9488, with an F value of 2064, R 2 -adjusted of 98.3% and σ ε1 = 1.391. Both values of a0 and a1 are significant. However, since t is measured in the order of 102 , assuming that a0 is different from zero would not appreciably improve the fit of the model (STDres is unchanged), as indeed was found by repeating the analysis with the above value of a0 . We therefore select a0 = 0, a1 = 1 (as initially assumed; refer to Equation (44)) and obtain the final estimates as given above with σε1 = 1.570. A scatter-plot of the residuals from this model is given in Figure 1. Note that although the application of this model required the estimation of three parameters, {α, λ, µ2 }, versus two parameters for all other models, the error, as evidenced by the value of STDres = 1471, is nearly halved relative to the best performing current model (model V). Also, the scatter of the residuals for the new model seems to be more random (no formal statistical analysis of randomness has been performed). Once all parameter estimates have been derived, the predicted value for observation i may be derived by introducing for the linear predictor, with its associated random deviation, the most probable value, namely ηi (the value of the linear predictor for observation i, with: z1 = 0). This is effectively the mean, E[log(Y )], used in the NL–LS part of the fitting procedure, if it is assumed therein that σε1 = 0 (refer to Equation (9)). Finally, approximate confidence intervals may be derived from the nonlinear regression procedure, as these are provided as a standard output from any statistical analysis package that performs nonlinear regression analysis.
6. MODELING A RANDOM RESPONSE WITH NO SYSTEMATIC VARIATION In developing the basic model in Section 3 (and in Appendix A) it was assumed that the two sources of random variation (self-generated and externally-generated) are independent (Assumption C). In this section, we first c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
74
H. SHORE
6000 4000 2000 10000 20000 30000 40000 50000 60000 -2000 -4000
6000 4000 2000 10000 20000 30000 40000 50000 60000 -2000
3000 2000 1000
-1000
10000 20000 30000 40000 50000 60000
-2000 -3000
Figure 1. Scatter plots of residuals (versus exact values of y), from fitting (top to bottom) models IV and V and the new model (Equation (8))
remove the restrictive assumption of independence and assume that both random errors derive from a bivariate normal distribution. We then assume that no systematic variation exists (namely, the linear predictor is constant) and examine the implications for modeling a response with a given (constant) error-distribution. Assume first that the random errors in (8) are derived from a bivariate normal distribution with correlation ρ. Specifically, let Z1 and Z2 be distributed according to a bivariate standard normal distribution with correlation ρ. The conditional distribution of Z2 , given Z1 = z1 , is then normal with mean ρz1 and variance 1 − ρ 2 and we may write, for Z2 in Equation (8), Z2 |z1 = ρz1 + (1 − ρ 2 )(1/2)Z2 c 2003 John Wiley & Sons, Ltd. Copyright
(47) Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
75
where Z2 is a standard normal variable, independent of Z1 . Introducing from (47) into (8) we finally obtain Y = exp{(α/λ)[(η + σε1 Z1 )λ − 1] + µ2 + (σε2 )[ρZ1 + (1 − ρ 2 )(1/2)Z2 ]}
(48)
where, for convenience, we replaced Z2 with Z2 , namely Z2 is now assumed to be a standard normal variable which is independent of Z1 . Assume, next, that no systematic variation exists, namely the linear predictor is constant. Without loss of generality we assume that η = 1. We obtain from (48) Y = exp{(α/λ)[(1 + σ ε1 Z1 )λ − 1] + µ2 + (σε2 )[ρZ1 + (1 − ρ 2 )(1/2)Z2 ]}
(49)
It is easily realized that for the extreme cases of ρ = 0 and ρ = 1, we obtain two expressions that have been formerly demonstrated to represent commonly used distributions well. For ρ = 1, we obtain Y = (M) exp{(α/λ)[(1 + σε1 Z1 )λ − 1] + (σε2 )Z1 }
(50)
where M is the median (Z1 = 0). Equation (50) is recognized as the ‘origin’ inverse normalizing transformation (INT), initially introduced in Shore11,10 and used in Shore11 to develop general schemes for control charts. This INT has been shown in Shore11 to represent many diversely-shaped distributions well, including gamma, Weibull, extreme value, Rayleigh and the normal and log-normal as exact special cases. For ρ = 0, we obtain Y = exp{(α/λ)[(1 + σε1 Z1 )λ − 1] + µ2 + (σε2 )Z2 }
(51)
This expression was also shown to represent many known and widely used distributions well12 . While for (50) easy-to-apply moment-matching estimation procedures have been developed for use in quality applications (refer to Shore11), Equation (51) requires numerically more complex fitting procedures and, therefore, is not recommended for use in actual applications. For the quality practitioner, a major implication of the above results is that the general model of Equation (8) (a special case of which is given by Equation (50)) can also be used to model a general response errordistribution, when the actual distribution is unknown or unspecified. The properties of the error distribution, implied by the RMM model (Equation (8)), are explored in some depth in Shore12.
7. CONCLUSIONS A general model for a response with both systematic and random components of variation has been shown to represent well the mainstream models developed over the years in various scientific and engineering disciplines, like physics, chemistry, biology, chemical engineering and electric engineering. Assuming that no systematic variation exists, the new model reduces to an INT, developed earlier from a different set of arguments11 and shown to represent well many of the distributions commonly used by quality practitioners. A major departure of the new modeling methodology from current methodologies is the introduction of two independent error terms, assumed to originate in different sources. The author of this paper is strongly convinced that this dual structure of the error is essential for a faithful description of any relationship between a response and the related ‘linear predictor’. For the quality discipline, the new approach seems to carry two important implications. Quality practitioners often face situations where they need to model a response without having the resources (in terms of either time or money) for an in-depth study of the scientific or engineering fundamentals that will allow proper theoretically-based modeling. We have shown that if the (unknown) relationship between the response and the ‘linear predictor’ can be assumed to be monotone convex (concave), the quality practitioner may conveniently use the new modeling approach, which is both distribution-free and structure-free, and expect a satisfactory goodness-of-fit. A second major implication is that for a non-normal environment, when no systematic variation c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
76
H. SHORE
is expected, the new model may provide a good representation for the unknown non-normal distribution, as indeed had been demonstrated in Shore11,12 . Future reported applications, in addition to those already published (see Shore1,8,11 ), would provide the cumulative evidence necessary to assess the relative merits and drawbacks of the new approach relative to current approaches.
REFERENCES 1. Shore H. Modeling a non-normal response for quality improvement. International Journal of Production Research 2001; 39(17):4049–4063. 2. Shore H. Modeling a response with self-generated and externally generated sources of variation. Quality Engineering 2002; 14(4):563–578. 3. Shore H. General control charts for attributes. IIE Transactions 2000; 32:1149–1160. 4. Lewis SL, Montgomery DC, Myers RH. Confidence interval coverage for designated experiments analyzed with GLMs. Journal of Quality Technology 2001; 33:279–292. 5. Myers RH, Montgomery DC. A tutorial on generalized linear models. Journal of Quality Technology 1997; 29(3):274– 291. 6. Hamada M, Nelder JA. Generalized linear models for quality-improvement experiments. Journal of Quality Technology 1997; 29(3):292–304. 7. Myers RH, Montgomery DC, Vining GG. Generalized Linear Models, with Applications in Engineering and the Sciences. Wiley: New York, 2002. 8. Shore H, Brauner N, Shacham M. Modeling physical and thermodynamic properties via inverse normalizing transformations. Industrial and Engineering Chemistry Research 2002; 41:651–656. 9. Shore H. Three approaches to analyze quality data originating in non-normal populations. Quality Engineering 2000; 13(2):277–291. 10. Shore H. Inverse normalizing transformations and a normalizing transformation. Advances in Methodological and Applied Aspects of Probability and Statistics, vol. 2, Balakrishnan N (ed.). Gordon and Breach Science: 2001; 131– 146. 11. Shore H. General control charts for variables. International Journal of Production Research 2000; 38(8):1875–1897. 12. Shore H. Response Modeling Methodology (RMM)—exploring the properties of the implied error-distribution. Communications in Statistics (Theory and Methods) 2002; 31(12):2225–2249. 13. Daubert TE. Evaluated equation forms for correlating thermodynamic and transport properties with temperature. Industrial and Engineering Chemistry Research 1998; 37(8):3260–3267. 14. Halliday D, Resnick R, Walker J. Fundamentals of Physics, Extended, with Modern Physics (4th edn). Wiley: New York, 1993. 15. Antoine C. Thermodynamic vapor pressures: New relation between the pressures and the temperatures (Thermodynamique, Tensions des Vapeurs: Novelle Relation entre les Tensions et les Temperatures), C.R. Hebd. Seances Acad. Sci. 1888; 107:681, 836, 1146. 16. Clapeyron EJ. Memoirs on the motive power of heat. L’ecole Polytechnique 1834; 14(23):153. 17. Riedel L. A new universal formula for vapor pressure (Eine Neue Universelle Dampfdruckformel). Chem. Ing. Tech 1954; 26:83. 18. Guggenheim EA. The principle of corresponding states. Journal of Chemical Physics 1945; 13:253. 19. Rackett HG. Equation of state for saturated liquids. Journal of Chemical and Engineering Data 1970; 15(4):514–517. 20. Zabransky M, Ruzicka V Jr., Majer V. Heat capacities of organic compounds in the liquid state. I. C1 to C18 1-Alkanols. Journal of Physical and Chemical Reference Data 1990; 19(3):719–762. 21. Benedek GB, Villars FMH. Physics with Illustrative Examples from Medicine and Biology—Statistical Physics (2nd edn). Springer: New York, 2000. 22. Schroeder M. Fractals, Chaos, Power Laws. W H Freeman and Company, 1991. 23. Goel AL, Okumoto K. Time-dependent error-detection rate model for software reliability and other performance measures. IEEE Transactions on Reliability 1979; 28:206–211. 24. Yamada S, Osaki S. Nonhomogenous error detection rate models for software reliability growth. Stochastic Models in Reliability Theory. Springer: Berlin, 1984; 120–143. 25. Musa JD, Iannino A, Okumoto K. Software Reliability—Measurement, Prediction, Application. McGraw-Hill: New York, 1987. c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
77
26. Duane JT. Learning curve approach to reliability monitoring. IEEE Transactions 1964; 2:563–566. 27. Xie M, Zhao M. On some reliability growth models with simple graphical interpretations. Microelectronics and Reliability 1993; 3(2):149–167. 28. Knafl GJ, Morgan J. Solving ML equations for 2-parameter Poisson-process models for ungrouped software-failure data. IEEE Transactions on Reliability 1996; 48:159–168. 29. Arnoux F, Gaudoin O, Makni C. The generalized power family in software reliability data analysis. MMR’2000: Second International Conference on Mathematical Methods in Reliability—Abstracts’ Book, vol. 1. 2000; 107–110. 30. Musa JD. Software Reliability Data. Technical Report, Rome Air Development Center, 1979. 31. Xie M. Software reliability models—past, present and future. Recent Advances in Reliability Theory—Methodology, Practice and Inference (Lecture Notes in Computer Science, vol. 1845), Limnios N, Nikulin M (eds.). Birkhauser: Boston, MA, 2000; 325–340.
APPENDIX A Five assumptions form the basis for the new modeling approach. Assumption A. The modeled response experiences self-generated random variation with a median ‘mass’ of M. The dispersion of the self-generated random deviations is proportional to M. This assumption is a natural outcome of our assertion that the self-generated variation reflects the inner instability of the response and, therefore, should be a function of M. A linear (proportional) relationship seems a plausible choice. Assumption B. Self-generated deviations are produced by a random mechanism that is driven by a normal random variable. Assumption C. The modeled response experiences externally-generated variation, which reflects variation transmitted to the response by the affecting factor (or a linear combination of factors, denoted ‘the linear predictor’). Deviations in the response that may be attributed to this source result from systematic variation, delivered by variation in the ‘linear predictor,’ and from random variation, delivered by an additive random error. The latter is modeled by a normal variable, assumed to be independent of the normal variable that accounts for self-generated variation. Assumption D. The affecting external agent (the linear predictor) will in most cases be an additive combination of the individual factors. Ignoring the self-generated variation, the quantile-relationship between the linear predictor (with the associated random error) and the response is monotone convex (second derivative is everywhere non-negative) or monotone concave. Throughout this paper, we only refer to a convex relationship, but it should be understood that any claim with regard to a monotone convex function is equally valid for a monotone concave function. Assumption E. Variation caused by the external source of variation interacts with the self-generated variation. This implies that self-generated and externally generated deviations mutually interact in determining the value of the response. Thus, a multiplicative model is assumed. We will now develop the general model for the quantile function of the response, Y , in terms of the two sources of variation. First, as implied by Assumption E, the response Y is modeled as Y = f1 (η, Z1 ; θ1 )f2 (Z2 ; θ2 )
(A1)
where f1 and f2 are as defined in Section 2. Let us now model f1 and f2 individually. We start by modeling f2 . By Assumptions A and B, the simplest model for self-generated random variation that increases with M is a proportional model f2 (Z2 ; θ2 ) = M(1 + σε2 Z2 ) = M + (Mσε2 )Z2
(A2a)
where: θ2 = {M, σε2 } is a vector of real-valued parameters that need to be determined. Since we may presume that for the relative random error |σε2 Z2 | 1 (assuming that Y is statistically stable), this relationship may be c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
78
H. SHORE
approximately represented by f2 (Z2 ; θ2 ) = M(1 + σε2 Z2 )[1/(σ ε2Z2)](σ ε2Z2) ≈ exp[log(M) + σε2 Z2 ] = exp[µ2 + σε2 Z2 ]
(A2b)
where θ2 = {µ2 , σε2 }, and µ2 and σε2 are the mean and the standard deviation of the normal variable that generates self-variation. In other words, we assume that the self-generated variation may be modeled by a log-normal variable. As is evident by the survey of current mainstream models (Section 4), both versions of Equations (A2) are used in scientific or engineering endeavors to model a given response. Thus, (A2b) is typical in modeling the log of the response (refer, for example, to Equation(4)), while (A2a) is the adopted model where the original response is modeled (Equation (5)). Next, we model the relationship f1 . By Assumption D, the relationship between the linear predictor (with the associated random component) and the response is monotone convex. However, one can conceive of many functions with this characterization. We therefore should ask ourselves whether the universe of all monotoneincreasing convex functions that can potentially model f1 (η; Z1 ) may be represented by a limited number of some fundamental types of functions, that in their totality may deliver good representations to all possible functions that conform to the above characterization (properties) of f1 ? To do this, we first present ‘The ladder of fundamental uniformly convex functions’. Since the main feature that distinguishes the various possible forms of f1 is the ‘tendency’ to curve upward (namely, how rapidly the function is bending upward with an increase in the ‘linear predictor’), we believe that any general model for f1 should be able to include all the cases of ‘the ladder’ (to be specified shortly) as special cases. Inspecting the models in Sections 2 and 4 and elsewhere2,8 , an outstanding observation is that there are indeed some fundamental convex functions that repeatedly make an appearance in the various models. These functions can be arranged in a hierarchical fashion, which starts with the linear and ends with the exponential– exponential–power relationship. We call this hierarchy ‘the ladder of fundamental uniformly convex functions’. It is presented below, where Z represents a standardized normal variable and V stands for the linear predictor with its associated random error, namely V = η + σ Z. The second derivative is also given to allow the reader assess how certain parameters determine whether an expression is uniformly convex or concave. 1. Linear increase. This is a convex relationship with the smallest rate-of-increase for the slope, representing a second derivative of zero: f1 = V = η + σ Z. 2. Power increase. This represents a monotone increase of the form V k , where k is a real-valued parameter. The second derivative is k(k − 1)V (k−2) . 3. Exponential increase. This represents a monotone increase of the form exp(V ). The second derivative is exp(V ). 4. Exponential–power increase. This represents a monotone increase of the form exp(V k ). The second derivative is kV (k−2) exp(V k ){(k − 1) + kV k } 5. Exponential–exponential increase. This represents a monotone increase of the form exp[(a) exp(V )], where the parameter ‘a’ is real-valued. The second derivative is a[1 + (a) exp(V )] exp[(a) exp(V ) + V ] 6. Exponential–exponential–power increase. This represents a monotone increase of the form exp[(a) exp(V k )]. The second derivative is (ak)V (k−1) exp[(a) exp(V k ) + V k ][(k − 1)/V + kV (k−1) + (ak)V (k−1) exp(V k )] This hierarchy is not uncommon in certain disciplines (like in calculating algorithmic complexity) and a general modeling of uniformly convex/concave relationships should be able to accommodate at least the functions included in the ladder of fundamental uniformly convex functions. A natural way to form such a unifying relationship that would represent all of the above as ‘special cases’ is by multiplying or adding the above functions, introducing enough parameters to allow the extraction of each c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
RESPONSE MODELING METHODOLOGY VALIDATING EVIDENCE FROM ENGINEERING AND THE SCIENCES
79
of the above cases individually. This approach, however, violates the important principle of parsimony, which requires that the underlying model be as simple as possible. An alternative approach needs to be pursued. Let us express f1 by f1 (η, Z1 ; θ1 ) = exp{(α/λ)[(η + σε1 Z1 )λ − 1]}
(A3)
where θ1 = {α, λ, σ ε1 } is a vector of real-valued parameters that need to be determined. Note that if there is systematic variation delivered by external factors to the response, this variation, transmitted via variation in η, will affect both the mean and the variance of the response. On the other hand, if no systematic variation exists, then f1 (η, Z1 ; θ1 ) represents the response as a random variable with a statistically stable distribution. It is easy to realize that all of the six cases included in the ‘ladder’ may be derived from (A3) by a proper selection of parameters. In particular, we obtain case 1 (the normal) by setting λ = 0, α = 1; case 2 by setting λ = 0, α = 1; case 3 by setting λ = 1; and case 4 by restricting λ = 1. Cases 5 and 6 are obtained if we re-introduce the linear predictor in the form of f1 , namely: we introduce, for η + σε1 Z1 in (A3), exp{(B/k)[(η + σε1 Z1 )k − 1]}. Obviously the values B = 1, K = 0, re-introduce the linear predictor as in (A3) while K = 0 introduces an exponential–exponential–power relationship. Since K = 0 represents an extreme case of convexity rarely encountered in practice, the above values of B and K may be generally presumed.
c 2003 John Wiley & Sons, Ltd. Copyright
Qual. Reliab. Engng. Int. 2004; 20:61–79
