A technique for early prediction of software reliability ... - Springer Link

Int J Syst Assur Eng Manag (Oct-Dec 2011) 2(4):261–281 DOI 10.1007/s13198-011-0078-1

ORIGINAL ARTICLE

A technique for early prediction of software reliability based on design metrics Sirsendu Mohanta • Gopika Vinod Rajib Mall

•

Received: 4 November 2011 / Revised: 12 December 2011 / Published online: 17 January 2012 Ó The Society for Reliability Engineering, Quality and Operations Management (SREQOM), India and The Division of Operation and Maintenance, Lulea University of Technology, Sweden 2012

Abstract In the early stages of development, it is difficult to quantitatively assess the reliability of a software product. In this context, we propose a bottom-up approach to predict the reliability of an object-oriented software from its product metrics gathered during the architectural design stage. A fault model is constructed to categorize different kinds of faults that can occur in the components making up the software product. Subsequently, the product metrics collected during the software design phase are used to estimate the expected number of different kinds of faults that may occur in a component. Eventually, these estimated values of the different kinds of faults are used to predict the expected values of the total number of faults present in the component. We use the estimated fault content of the component and the number of tests that will be performed over the component, to predict reliability of the component. We adopt a probabilistic approach, Bayesian Belief Network, for reliability prediction of the components from product metrics. Based on predicted reliabilities and usage frequencies of the components, the reliability of a system is predicted. The applicability of our proposed model is illustrated through a case study. Moreover, we performed a set of experiments and also compared our approach with an established approach reported in the literature to investigate the accuracy of our S. Mohanta (&) R. Mall Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur 721302, India e-mail: [email protected] R. Mall e-mail: [email protected] G. Vinod Bhabha Atomic Research Centre, Mumbai 400085, India e-mail: [email protected]

approach. Analysis of the results from our experiments suggests that our approach yields reasonably accurate result. Keywords Software reliability prediction Fault model Design metrics Bayesian Belief Network

1 Introduction Safety-critical applications, such as aviation, nuclear power generation, satellite communication are required to be highly reliable as failure of these systems may cause injury or death to human beings. Apart from safety-critical systems, software has become the integral part of most of complex applications. Thus it is very important to ensure that the underlying software will perform its intended functions properly. Hence, software reliability analysis has become a heavily researched area (Bai et al. 2005; Cai et al. 2008; Cheung et al. 2008; Cortellessa et al. 2002; Gokhale and Trivedi 2002; Huang 2005; McCall et al. 1992; Reussner et al. 2003; Tripathi and Mall 2005). In general, the software reliability related research studies that have been reported in the literature can be classified into reliability estimation approaches and reliability prediction approaches. Reliability estimation approaches (Ball et al. 2005; Cai et al. 2008) quantify the reliability of a software application by fitting the failure data collected during testing or maintenance phase of the product development cycle to an appropriate growth model (Huang 2005; Inoue and Yamada 2010). As a result, it becomes too late and costly to initiate any corrective actions based on estimated reliability figures by the reliability estimation approaches. Another shortcoming of reliability estimation approaches is that in general, these approaches treat a software system as a monolithic

123

262

element. They ignore the internal structure of the system, and thus are known as black-box approaches. Therefore, these approaches are difficult to use when relating the overall system reliability to the reliability of its constituent components. Reliability prediction approaches (Cheung et al. 2008; Cortellessa et al. 2002; Gokhale and Trivedi 2002; McCall et al. 1992; Reussner et al. 2003; Tripathi and Mall 2005) directly leverage the reliability of individual components and their architectural configuration to predict reliability of the overall system during early stages of product development, and are known as white-box approaches. Most of these reliability prediction approaches assume that a system is composed of several components of known reliability values. Also the failures of various components are assumed to be independent events. In other words, it is implicitly assumed that reliability of one component is not influenced by other components. However, both of these assumptions are far from being realistic. Reliability figures of the components are usually not available during the early stages and it is well accepted that various components of a system directly or indirectly influence each other through control or data flow relationships. In this context, we propose a bottom-up approach named early software reliability assessment (ESRA) for reliability prediction of an object-oriented software during the early stages of product development. The choice of target software systems is primarily motivated by the increasing commercial and academic importance of the object-oriented paradigm. The proposed ESRA works mainly in three phases: class reliability prediction, use case reliability prediction and system reliability prediction. The classes in the object-oriented system are considered as the basic components in ESRA approach and the reliabilities of the classes are predicted first. A fault model is constructed to categorize different kinds of possible faults that can occur within a class. Subsequently, a number of product metrics that have strong bearing on the different categories faults are identified and used to estimate the expected number of different types of faults in the class in order to compute total fault content of the class. Reliability of a class is determined based on the fault content present in the class and the number of times the class will undergo the testing and debugging process. In this study, a probabilistic approach namely Bayesian Belief Network (BBN) is adopted to predict reliabilities of the classes from product metrics. The reliability of a use case is predicted by analyzing the operational profile and the reliabilities of classes whose objects are involved in execution of the use case scenarios. Moreover, in use case reliability computation, error propagation (EP) among the classes are considered. Finally, the system reliability is predicted based on the use case reliabilities and the operational profile (Musa 1993).

123

Int J Syst Assur Eng Manag (Oct-Dec 2011) 2(4):261–281

The rest of this paper is organized as follows. In Sect. 2 we discuss some basic concepts on which our approach is based. Section 3 presents our proposed approach for early prediction of software reliability. We discuss a case study to illustrate the applicability of our proposed approach in Sect. 4. In Sect. 5, we perform experimental studies to evaluate our approach and discuss the results. We compare our approach with related work in Sect. 6. Finally, Sect. 7 concludes this paper.

2 Background concepts In this section, we discuss a few basic concepts that have been used in our work. First we provide a brief overview of the BBN. Subsequently, we discuss EP in software architectures based on the work of Abdelmoez et al. (2004). Finally, we give a brief overview of the object-oriented product metrics and a few relevant UML diagrams. 2.1 Bayesian Belief Network A BBN is by a directed acyclic graph (DAG). Each node (or vertex) of this graph represents a discrete or continuous random variable. The directed edges represent causal dependencies among variables. When two nodes are connected by a directed edge, the node at the tail of the edge is called the child and the other as parent. The nodes having only outgoing edges are called root nodes. The variable represented by a root node is not dependent on any other variable. The dependency between two variables is quantified by the conditional probability distribution. Formally, a Bayesian network is defined as the set {D, S, P}, where – –

–

D is a set of random variables, D ¼ fD1 ; D2 ; . . .; Dn g: Each Di in D can either be discrete or continuous. S is a set of conditional probability distributions. S ¼ fpðD1 jParentsðD1 ÞÞ; . . .; pðDn jParentsðDn ÞÞg; where, Parents(Di) denotes all the parent nodes for Di, p(Di|Parents(Di)) is the conditional distribution of Di with respect to its parent nodes. P is the set of probability distribution of the random variables. P ¼ fpðD1 Þ; . . .; pðDn Þg where, p(Di) represents the probability distributions of random variable Di.

A BBN is applied to model causal influences, where the entities of a system are modeled as nodes of the BBN and the edges represent cause-effect relationships among the entities. BBN model-based solution to a problem is achieved using the following three steps. (1)

Construction of DAG: In this step, the independent and dependent variables of a given problem are identified. A node is drawn to represent the each


(2)

(3)

263

corresponding variable of the given problem. Later, these nodes are linked using the directed edges based on the dependencies among the variables. Assignment of probability distributions: To perform quantitative analysis with the BBN model, a probability is assigned to each random variable which is represented by a node in the constructed DAG. For this, a conditional probability density (CPD) is assigned to each node which corresponds to a continuous random variable and a node probability table (NPT) is assigned to each node which represents a discrete random variable. An NPT contains conditional probability of a variable given the states of its parents. The CPD or NPT for a node can either be derived from experimental observations or elicited from experts. Inference: Once the BBN model is constructed, the joint probability of all variables in the BBN is determined using the Bayesian theory. The probability of the dependent variable of concern in the BBN model is determined by marginalizing of the joint probability of all variables in the BBN. The probability computation of a node is known as probabilistic inference.

2.2 Estimation of error propagation (EP) in software architectures

– –

– –

Under the following two assumptions: (1) (2)

the EP probability EPðA ! BÞ from a component A to another component B can be computed using the equation as follows: P P 2 1 jS jjV1 j2 x2SB y2SB jFx1 ðyÞj B A!B EPðA ! BÞ ¼ ð2Þ 1 1 VA!B

–

EPðA ! BÞ P P 1 x2SB PB ðxÞ y2SB PA!B ½Fx1 ðyÞP0A!B ½Fx1 ðyÞ P ¼ 1 v2VA!B PA!B ½vP0A!B ½v ð1Þ where

–

Probability of state changes or state transitions of B are uniform All the messages passed from A to B are equiprobable

where

Several studies (Abdelmoez et al. 2004; Cortellessa and Grassi 2007; Mohamed and Zulkernine 2008) on estimation of the EP between software components have been reported. In the current study, we have adopted the EP model of Abdelmoez et al. (2004), which is briefly described in the following. The method proposed in Abdelmoez et al. (2004) shows that in the early stages of product development cycle, EP between software components can be calculated from the information about the flow of control and data among the components. In this current study, UML state transition diagrams of the components are used to compute the EP between components. The error propagation (EP) probability EP(A, B) from a component A to another component B can be quantified with help of the following equation.

–

Fx1 ðyÞ ¼ fv 2 VA!B jFx ðvÞ ¼ yg: F represents the state transition mapping, i.e. F:SBVA!B ! SB PB(x) is the probability of observing component B in state x (x 2 SB ) and SB is the set of all states of the component B. PA!B is the probability of a messages v 2 VA!B getting correctly transferred from A to B. P0A!B is the probability of incorrect messages being transferred from A to B thereby making component B to transit to an incorrect state. Here, messages are generic in the sense that they can be in the form of data, function parameters etc.

VA!B is the set of messages transferred from component A to B. v is an item in the set VA!B :

– –

|SB| is the cardinality of the set of all states of the component B. jVA!B j is the cardinality of the set of all messages transmitted from A to B. |F-1 x (y)| is the number of messages from A that make B change its state from x to y.

In the early phases of product development cycle, the information necessary to determine F may not be available. Therefore, Abdelmoez et al. (2004) used the following equation to estimate an upper bound of EPðA ! BÞ: EPðA ! BÞ

1 jS1B j 1 1 jVA!B j

ð3Þ

Moreover, EPðA ! BÞ represents the conditional probability of an error propagating from component A to an other component B, given that A transmits messages to B. If a system consists of N components, EP probability can be represented by an N 9 N matrix, where the entry at Ath row and Bth column is the EP probability from component A to component B and EPðA ! AÞ is equal to 1 (Abdelmoez et al. 2004). The Unconditional EP denoted by EðA; BÞ is defined as the product of the conditional EP probability and the probability of message transfer from one component to another. EðA; BÞ is expressed as follows:

123

264


EðA; BÞ ¼ EPðA ! BÞ TA; B

ð4Þ

where T is the N 9 N transmission probability matrix whose entry TA, B is the probability that the component A sends a message to component B. In other words, TA, B is the transition probability which indicates the probability of the connector from A to B to be activated during execution; where, X TA; A ¼ 0 and TA; B ¼ 1: ð5Þ

Table 1 Description of the object-oriented design metrics used in our study Name of the metric

CK metrics suite (Chidamber and Kemerer 1994) Weighted Methods per Class (WMC)

This metric is defined as the sum of the complexities of all methods in a given class

Coupling Between objects (CBO)

It counts the number of classes whose attributes or methods are used by the given class and the number of classes which use the attributes or the methods of the given class

Depth of Inheritance Tree (DIT)

It is defined as the length of the longest path from a given class to the root class in the inheritance hierarchy

Lack of Cohesion of Methods (LCOM)

It counts the sets of methods in a class that are not related through the sharing of some local variables of the class

Number of Children (NOC)

It is the count of the total number of immediate child classes of a given class

Response for Class (RFC)

It measures the number of methods (NOM) and constructors that can be invoked as a result of a message sent to an object of the class

B

2.3 Object-oriented product metrics In the past, several research results (Abreu et al. 2002; Bansiya and Davis 2002; Chidamber and Kemerer 1994; Genero et al. 2002) have been reported in the literature concerning object-oriented product metrics available during design state of the product development. The metric suites proposed by Chidamber and Kemerer (1994) and Abreu et al. (2002) have frequently been referred. These two metrics suites are titled as CK (Chidamber and Kemerer) and MOOD (metrics for object-oriented design) suites respectively. In addition to CK and MOOD metric suites, QMOOD (quality MOOD) by Bansiya and Davis (2002) has also been reported in the literature. Genero et al. (2002) reported a set of design metrics for UML statechart diagrams. A brief description of the objectoriented product metrics is presented in Table 1.

Description

MOOD metrics suite (Abreu et al. 2002) Method Inheritance Factor (MIF)

It represents the ratio of inherited methods to the total NOM in a class

Attribute Inheritance Factor (AIF)

It represents the ratio of inherited attributes to the total number of attributes in a class

QMOOD metrics suite (Bansiya and Davis 2002)

3 ESRA: our early reliability prediction approach In this section, we present our ESRA approach to predict the reliability of an object-oriented software. The proposed ESRA approach which is schematically shown in Fig. 1, works in three phases. In the first phase, the reliabilities of the classes are predicted from the design metrics extracted from UML model of the software application. This phase is known as class reliability prediction. The second phase which is known as use case reliability prediction involves prediction of use case reliabilities based on the operational profile and predicted reliabilities of the classes whose objects are involved in the executions of use case scenarios. In the last phase, the system reliability is predicted from reliabilities of the use cases and the operational profile. Hence, the last phase is known as system reliability prediction. In the following sections, the three phases of ESRA approach is explained in more detail. 3.1 Phase 1: class reliability prediction During the development of a class in an object-oriented application system, several types of faults may creep in the

123

Data Access Metric (DAM)

This is represented by the ratio of private and protected attributes to the total number of attributes declared in a class

Class Interface Size (CIS)

This is a count of total number of public methods of a class

Direct Class Coupling (DCC)

A count of classes that accepts instances of a given class as a parameter and the classes that include attributes of the given class type

Statechart metrics (Genero et al. 2002) Number of Transitions (NT)

This metric measures the total NT in the state diagram of a given class

Number of States (NS)

This metric computes the total NS in the UML state diagram of a given class

Number of Activities (NA)

This metric counts the total number of events in the UML state diagram

Number of Entry Actions (NEntryA)

The total number of the actions performed each time a state is entered

Number of Exit Actions (NExitA)

The total number of actions performed each time a state is left

class. We categorize these different kinds of faults and build a hierarchy of fault types called as fault model. Subsequently, the product metrics that correlate significantly with specific categories of faults are identified. The values of identified product metrics are computed from


265

Legend Numerical Value

Artifacts

Functional blocks

Flow

Phase 1: Class Reliability Prediction

Operational Profile Class Reliabilties

Operational Profile

Phase 2: Use Case Reliability Prediction

Use Case Reliabilties

Phase 3: System Reliability Prediction

System Reliability

UML Model (XML Format)

Fig. 1 Schematic representation of ESRA approach

UML diagrams of the system. Subsequently, a DAG for a BBN model is constructed utilizing information of the fault model and the product metrics. Finally the computed product metrics are used as the inputs to the constructed BBN model to predict reliabilities of the classes as the output. The predicted class reliability values are used as the input to phase 2 or use case reliability prediction. These steps for class reliability prediction has been schematically shown in Fig. 2. In the following sections, we discuss the each step involved in the class reliability prediction phase in more detail. 3.1.1 Fault model construction In this study, we extend our proposed fault model reported in Mohanta et al. (2010). Figure 3 shows that at a very high level, faults that can occur in a class can be classified into requirements faults, design faults, programming faults and traceability related faults. As it goes beyond the scope of this paper to describe each type of faults in detail, we limit ourselves to exemplify a few of these faults. Requirements faults can occur due to human errors, lack of understanding of system functionalities, misunderstanding between the system analyst and customer, etc. A

few examples of requirements faults are discussed in the following. – Missing of functional specification: This kind of error can occur when the system analysts overlooks to specify or document a functional requirement of a system during requirements analysis phase. – Incorrectly specified requirements: This occurs when a system analyst incorrectly specifies or documents a functionality of a system. Design faults are closely related to human factors and the design process (Lanza et al. 2005). A few examples of design based faults can be found in Lanza et al. (2005). Programming faults arise due to human errors and a programmer’s unfamiliarity with the programming language constructs. Programming faults can further be classified into structural faults, and algorithmic faults. Structural faults can be divided into procedural faults and object-oriented faults types. Procedural faults are those that a programmer commits due to large program size, overly complex program structure, complex data-flows, complicated control-flows etc. Procedural faults can further be classified into four types: control and sequence faults, computational faults, logical faults, initialization faults. A few examples of control and sequence faults are the following:

BBN Construction

Fault Model

BBN

Design Metrics Computation

Design Metrics

Class Reliability Prediction

Class Reliabilities

Phase 2: Use Cae Reliability Prediction

Phase 1: Class Reliability Prediction

UML Model (XML Format)

Legend

Flow

Has an Impact

Numerical Value

Artifacts

Functional blocks

Fig. 2 Schematic representation of class reliability prediction

123

266


Fig. 3 A fault model for a class Faults

Requirements Faults

Programming Faults

Design Faults

Traceability related Faults

Algorithmic Faults

Structural Faults

Object-oriented Faults

Inheritance related Faults

Procedural Faults

Polymorphic Faults

State-transition related Faults

Control and Sequence Faults

–

–

Non-terminating loop: In this type of fault, a loop does not terminate possibly because the loop conditions are incorrectly formed. Improper nesting of loops: This type of faults can occur when the nested loops have incorrectly been nested.

Computational faults include the faults occurring on account of incorrect conversion from one data representation to another. A few examples of computation faults are described in the following. –

–

Round off error: This kind of faults occurs when a value to be stored in a variable exceeds the largest value that the data type of the variable can store. Incorrect bounds: This kind of faults are caused by an access to an array outside of its specified bound.

Logical faults occur due to incorrect use of logic operators in a program. In the following, we mention two examples of logical faults. – Operator precedence error: This kind of faults occur when a programmer either omits required parentheses in an arithmetic expression, or inserts parentheses at a wrong place. – Improper conditional expression: This type of faults is caused by inappropriate use of logical operators in a decision construct. Initialization faults are caused due to improper or superfluous initialization of variables, loop-control variable or function parameters. An example of initialization faults is described in the following.

123

–

Computational Faults

Requirements to design traceability fault

Logical Faults

Design to code traceability fault

Initialization Faults

Incorrect initialization: This category of faults occurs when a programmer either initializes a variable with incorrect value or forgets to initialize a variable with any value at all.

Apart from procedural faults, object-oriented features, such as inheritance, polymorphism, abstraction etc. cause additional types of faults. We have named these category of faults as object-oriented faults. A few number of faults can occur due to incorrect implementation of UML state model, we have named it as sate-transition related faults. A few examples of inheritance, polymorphic and statetransition related faults are discussed in Binder (1996), Mohanta et al. (2010), Offutt et al. 2001). Algorithmic faults arise either due to use of an incorrect algorithm or an incorrect understanding of the problem for which the algorithm is designed. A few examples of algorithmic faults are the following. –

–

Inappropriate algorithm selection: This type of faults occur when a programmer selects an incorrect algorithm for the problem. Unsatisfactory performance: This type of faults can occur when a programmer implements a suboptimal algorithm which results in unsatisfactory performance of the program.

Traceability related faults occur when a software developer misinterprets the requirements specification or design. Traceability related faults can further be classified into requirements to design traceability fault and design to


code traceability. A few examples of traceability related faults are discussed in Mohanta et al. (2010) 3.1.2 Identification of design metrics correlated to specific faults In the recent past, a large number of studies (Fenton and Neil 1999; Malhotra et al. 2010; Olague et al. 2007) have been carried out to explore the relationship among software design metrics and software code faults only. However, in the literature, work exploring the relationship among software product metrics and different categories of faults has scarcely been reported. In order to explore the relationship among various design metrics and different categories of faults, bivariate correlation analysis is performed. The CK (Chidamber and Kemerer 1994) metrics, MOOD (Abreu et al. 2002), QMOOD (Bansiya and Davis 2002) and the metrics suite proposed by Genero et al. (2002) are considered for the correlation study. Among the various metrics in CK, MOOD, QMOOD metrics suites, only those metrics which can be computed in the scope of a single class have been selected for the current study. The measure of functional abstraction (MFA), NOM of QMOOD metrics suites have been excluded from our study. This is because MFA metric is functionally equivalent to MIF of MOOD metrics suite and NOM is same as WMC of CK metrics suite when weights of the methods in the class equal to unity. A brief description of the product metrics that have been selected for this study is shown in Table 1 of Sect. 2.3.

267

We performed non-parametric Spearman’s rank correlation, with a level of significance a = 0.05 (two-tailed); which means the level of confidence is 95%. The current correlation study was limited to explore the relationship among various object-oriented metrics and design, programming and traceability related faults. Correlation analysis among object-oriented metrics and requirements faults, algorithmic faults is kept as scope of our future work. The results of our correlation study is shown in Table 2. Several guidelines for the interpretation of a correlation coefficient has been reported in the literature. However, all such interpretations of a correlation coefficient depends on the context and the purposes. The criteria that has been used in the current study to interpret the correlation coefficient is shown in Table 3. We only considered those product metrics that show either medium or strong correlation with specific category of faults. As shown in Table 2, results of our correlation study indicate WMC, CBO, RFC and DIT of CK metric suite and NS, NT metrics are correlated with design faults. Therefore, WMC, CBO, RFC, DIT, NS and NT metrics can be individually used to predict the design faults. Again, from Table 2, it can be observed that, WMC, CBO, RFC of CK metrics suite and CIS, DAM of QMOOD metrics suite are correlated with control and sequence faults. Hence, WMC, CBO, RFC, CIS and DAM can be used to predict control and sequence faults. Logical faults correlates with WMC, CBO, RFC of CK metrics suite and CIS of QMOOD metrics suite. Therefore, logical faults can be predicted

Table 2 Spearman’s correlation coefficients among CK, MOOD and QMOOD metrics and different types of faults Design faults

Control and sequence faults

Logical faults 0.439

Computational faults 0.431

Initialization faults 0.617

Inheritance and polymorphic faults 0.270

State-transition based faults 0.039

Traceability related faults

WMC

0.411

0.531

0.378

NOC

0.502

0.142

0.269

0.227

0.159

0.587

0.168

0.102

DIT

0.427

0.214

-0.121

0.214

0.172

0.602

0.122

0.017

CBO

0.511

0.575

0.483

0.602

0.532

0.121

-0.083

0.331

RFC

0.129

0.492

0.521

0.523

0.492

-0.092

0.152

0.274

LCOM

0.131

0.131

0.143

0.489

0.212

0.110

0.213

0.112

CIS

0.247

0.562

0.611

0.613

0.521

0.172

-0.124

0.024

DAM

0.061

0.612

0.313

0.421

0.107

0.215

0.102

0.002

DCC

0.117

0.149

-0.102

0.142

0.130

0.124

0.102

0.019

AIF

-0.103

0.251

0.112

-0.154

0.012

0.202

0.182

0.211

MIF NS

0.172 0.441

0.213 0.122

0.141 0.213

0.187 0.142

0.172 0.147

0.501 0.121

0.271 0.521

0.137 0.473

NT

0.326

0.205

0.152

0.172

-0.271

0.270

0.546

0.452

NA

0.177

0.157

-0.241

0.187

0.197

0.134

0.278

0.208

NEntryA

0.004

0.212

0.103

-0.280

0.148

0.231

0.214

0.102

NExitA

0.109

-0.131

-0.102

0.197

0.215

0.152

0.149

0.093

123

268


Table 3 Interpretation of correlation coefficients Strength of the correlation

Value of correlation coefficient Negative

Positive 0.0 to 0.09

None

-0.0 to -0.09

Small

-0.1 to -0.3

0.1 to 0.3

Medium

-0.3 to -0.5

0.3 to 0.5

Strong

-0.5 to -1.0

0.5 to 1.0

with the help of WMC, CBO, RFC and CIS metrics. We found WMC, CBO, RFC, LCOM, of CK metrics suite and CIS, DAM of QMOOD metrics suite are correlated with computational faults. Results in Table 2 also indicate that WMC, RFC, CBO of CK metrics suite and CIS of QMOOD metrics suite are correlate with initialization faults. Therefore, WMC, RFC, CBO and CIS metrics can be used individually to predict the initialization faults. Again, we also find a strong correlation among number of inheritance and polymorphic related faults and NOC, DIT of CK metrics suite and MIF of MOOD metrics suite. NS and NT metrics strongly correlate with state-transition related faults. Table 2 shows that traceability related faults is correlate with WMC, CBO, NS and NT metrics. 3.1.3 Construction of BBN model and class reliability prediction In this section, we explain the construction of a BBN model and computation of the class reliability from the object-oriented product metrics that have been identified in the Sect. 3.1.2. A BBN model is constructed through three steps: construction of DAG, assignment of probability distributions, and inference. In the following we discuss these steps in details. (1)

Construction of a DAG: A node is drawn for each category of faults and corresponding design metrics which correlate with each fault category. To represent the dependency relations among different metrics and number of different categories of faults in a class, we draw directed edges from the nodes representing software metrics to the nodes representing various categories of faults. The total number of faults in a class can be determined by aggregating the different categories of faults that creep into the class during development process. To represent this relation, a node labelled Total number of faults is added and directed edges are drawn from the nodes representing different categories of faults to the node labelled Total number of faults. Most of the reliability estimation approaches (Bai et al. 2005; Cai et al. 2008; Huang 2005) follow the

123

(2)

idea that the probability of failures of a software, that may eventually occur in the future time, can be estimated based on the number of faults present in a software application and the number of tests that will be performed over the software application. The same idea is adopted in our class reliability prediction. The reliability of a class is estimated based on the expected number of faults present in the class and the number of tests that the class will undergo. This relationship is modeled by drawing two nodes named Reliability of class and Number of tests and two edges from nodes Total number of faults, Number of tests to Reliability of class. The constructed DAG is shown in Fig. 4. After the DAG construction, the conditional probabilities are assigned to the individual nodes of this DAG in the subsequent step. Assignment of probability distributions: After structuring the BBN model, we need to specify the prior distributions of the nodes of the BBN. Each node in the constructed DAG represents either a continuous or discrete random variable, depending on their nature and the values they assume. A CPD is associated with node which represents continuous random variable. On the other hand, a NPT which contains the conditional probability distribution is associated with node which indicates a discrete random variable. The nodes with no parents are associated with the unconditional probability distributions. In many situations, a parametric form or functional from is used to specify the CPD for a child node. Some of these functions can be deterministic and some other can be probabilistic. We use generalized linear model (GLM) to relate the parent nodes indicating product metrics and child nodes representing the specific category of faults. If we assume Y is a dependent random variable and varies as a function of a set of independent predictor or regressor variables X ¼ X1 ; X2 ; . . .; Xk ; in GLM we have:

EðYÞ ¼ l ¼ g1 ½f ðX; bÞ

ð6Þ

where, Y is the set of observations, with expected value E(Y) = l, and g is the link function which is determined by the distribution of Y. f ðX; bÞ is the prediction function and b is the set of parameters. To specify the CPD for a child node in our constructed DAG, if we use linear predictor and an identity link function in the GLM, the mean of the child node distribution can be expressed as the weighted average of the values of the corresponding parent nodes. Thus, for a child node Y with parents X ¼ X1 ; X2 ; . . .; Xk weighted by b ¼ b1 ; b2 ; . . .; bk ; we have


269

Nodes representing independent variable Nodes representing dependent variable CBO

CIS

Links

DAM

Legend

RFC

NOC DIT

Number of logical faults

MIF

WMC Number of design faults

Number of control and sequence faults

LCOM

Number of computation al faults

NT

Number of inheritance and polymorphic faults

Number of initialization faults

Number of traceability related faults

NS

Number of tests Total Number of faults Number of state model based faults

Reliability of class

Fig. 4 BBN model for predicting the reliability of a class

EðYÞ ¼ l ¼

k X

bi Xi ¼ bX

ð7Þ

i¼1

pðYjXÞ N ðl; r2 Þ

ð8Þ

where, p(Y|X) is the CPD of Y with variance r2. Equation 7 is equivalent to multiple linear regression. Therefore, we can perform regression analysis to compute the set of weights b when sufficient data is available. When sufficient data is not available for performing regression, the weights are elicited subjectively. The above idea can be used to determine the CPD for a specific category of faults. The CPD for a specific category of faults can be determined by performing a multiple linear regression analysis over different categories faults and corresponding design metrics that correlate strongly with the fault category. For example, CPD of number of inheritance faults can be expressed by a linear regression equation, in which number of inheritance faults is the dependent variable and various inheritance metrics, such as NOC, DIT, MIF are the independent variables. To estimate the coefficients (b) and r, we perform a maximum likelihood estimation. The CPD for child node labelled Total number of faults can be specified by a deterministic function which can be determined by aggregating Number of logical faults,

Number of computational faults, Number of control and sequence faults, Number of initialization faults, Number of traceability faults, Number of inheritance and polymorphic faults and Number state model based faults. To relate expected total number of faults present in a class and reliability of the class, we adopt the methodology that was proposed by Bishop (2002). According to their work, we can find the reliability of a class in the worst case using a few assumptions. These assumptions are discussed in the following. –

–

The individual failure rates ðk1 ; k2 ; . . .; kn Þ for N residual faults in a software product do not change with time and are unaffected by defect removal process. Fault detection and correction is immediate after any failure discovery.

Under the above two assumptions, for any defect i which has failure probability per test execution of ki, the probability of failure after T prior test executions is given by: PðTi ¼ T þ 1jki Þ ¼ ki ð1 ki ÞT

ð9Þ

where, Ti is test where a failure is observed and defect i is detected and removed. Although, we do not know the values of ðk1 ; k2 ; . . .; kn Þ; but we can find the a value of ki

123

270


after T tests such that failure probability P(T | ki) will be maximum. Differentiating the Eq. 9 with respect to ki, it is observed that the maximum failure probability occurs when: ki ¼

1 T þ1

ð10Þ

The maximum value of failure probability for defect i is: PðTi ¼ T þ 1Þ\

1 T e

ð11Þ

where, e is Euler’s number, whose value is approximately 2.7182. The work in Bishop (2002) suggests that, the failures of all faults are disjoint. Therefore, the maximum failure probability HðTÞ for N faults after T tests will be: HðTÞ\

N T e

ð12Þ

The worst-case reliability value after T tests can be given by: Rw ðTÞ ¼ 1

N eT

ð13Þ

where, Rw(T) is worst-case reliability value after T number of tests. N is the expected number of faults. CPD for the node labelled Reliability of class is deterministic and is given directly from Eq. 13. Number of tests indicates the number of times the functionalities of a class is going to be tested. Therefore, the probability distribution of node labelled Number of tests can be specified from the test plan and budget or from previous historical testing data of similar projects. Probability distributions of the nodes which represent the different metrics is assigned available metrics data or directly from the measurement. Once the probability distributions are associated with all nodes of the proposed BBN model, we use the model to predict the reliability of a class using the following inference mechanism. (3) Inference: In this step, we assign values to the independent variables of the BBN model to determine the probable values of the dependent variables. This probability computation of the random variable of concern is known as probabilistic inference. In the literature, researchers have proposed various algorithms (Lauritzen and Spiegelhalter 1998; Paul and Michael 1993) to determine inference propagation in BBN models. Several numerical techniques (Dagum and Chavez 1993; Yuan and Druzdzel 2007) have also been reported to compute the inference propagation in BBN models. Many BBN tools are available to construct BBN model and these tools use their inference propagation algorithm to determine the

123

probability value of the desired nodes of the BBN model. In the proposed BBN model, design metrics of a class are the independent variables and the reliability of the class is the dependent variable. A set of design metrics such as, WMC, RFC, DIT, NOC, MIF, NOM, CIS, APM, DAM, NA, NS, NT are computed from the UML diagrams. The values of these metrics are given as the input to the constructed BBN and reliability of the class is predicted from our constructed BBN with the help some inference propagation algorithms or belief updation algorithms (Dagum and Chavez 1993; Lauritzen and Spiegelhalter 1998; Paul and Michael 1993; Yuan and Druzdzel 2007).

3.2 Phase 2: use case reliability prediction A use case typically consists of multiple scenarios. A scenario represents a possible sequence of steps in the execution of a use case. The reliability of execution of a scenario would depend on the reliabilities of classes whose objects are involved in its execution. However, the reliabilities of classes that are involved in the executions of a use case scenario can be influenced by the EP from the other faulty classes. Hence, reliability of execution of the use case scenario depends on the reliabilities of the classes and the probability of EP among the classes. As shown in Fig. 5, first the probability of the EP among the classes is computed from the UML statechart specification and massages passing information. The probability of EP is computed based on the method discussed in Sect. 2.2. Subsequently, the probabilities of EP among the classes are used to refine the reliabilities of the classes involved in the execution of a use case scenario. The modified reliability jth class, R0w ðClji Þ; can be deterined with help of following equation: 0 1 P Y B C 1 Eðp; jÞ 1 Rw ðClpi Þ C R0w ðClji Þ ¼ Rw ðClji ÞB @ A p¼1 p6¼j

ð14Þ where, P number of classes interact to implement the ith use case scenario, Rw(Clji), Rw(Clpi) is the reliabilities of any two classes involved in ith use case scenario and Eðk; jÞ is the unconditional EP matrix which is defined in Sect. 2.2. The reliability of execution of a use case scenario, Rw(Sik), is computed utilizing the reliabilities of the classes that interact to implement ith use case scenario as follows:


Phase 3: System Reliability Prediction

Use Case Reliabilities

271

Failure Propagation Computation

Use Case Reliability Prediction

Operational Profile

Error Propagation Probability State Diagram

Class Reliabilities

Scenario Reliability Prediction

Scenario Reliability

Phase 1: Calss Reliability Prediction

Phsae 2: Use Case Reliability Prediction Legend Has an Impact

Flow

Numerical Value

Artifacts

Functional blocks

Fig. 5 A schematic representation of use case reliability prediction

Rw ðSik Þ ¼

P Y

R0w ðClji Þ

ð15Þ

Rw ðSysÞ ¼ 1

j¼1

ð17Þ

k¼1

After reliability prediction of the use case scenarios, the reliability of a use case is estimated from the predicted reliabilities of the use case scenarios using following expression: RðUk Þ ¼ 1

K Y ðð1 Rw ðUk ÞÞ Pk Þ

M Y ðð1 Rw ðSik ÞÞ pi Þ

ð16Þ

i¼1

where, Rw(Sik) is the reliability of the ith execution scenario of kth use case and M is the number of scenarios within the kth use case. Usage frequency of all scenarios (pi) can be determined from the operational profile (Musa 1993). 3.3 Phase 3: system reliability prediction A system can be modeled as a set of use cases. Consequently, the reliability of a system can be predicted from the predicted reliabilities of its use cases in phase 2 and their respective execution frequencies as shown in Fig. 6. If a system consists of K use cases and reliability of each use case is Uk (as predicted using Eq. 16), system reliability Rw(Sys) can be expressed as follows: Fig. 6 A schematic representation of system reliability prediction

where, Uk is the reliability of kth use case, and Pk is the execution probability of the kth use case. The execution probability of a use case can be determined from the operational profile (Musa 1993). To summarize our proposed ESRA approach, we presented a pseudocode describing steps that are involved in our approach in Algorithm 1. The figure shows the procedure ESRA uses the UML architectural models comprising of the use case, sequence, class and state diagrams of a software system and operational profile as the input. The first step (line number 2 in procedure ESRA) constructs a fault model for a class in an object-oriented system. A fault model consists of different types faults that can occur in a class. Once the fault model is constructed, a set objectoriented product metrics which correlate with a specific type of faults in the constructed fault model are identified. In line number 4, a BBN model is constructed for reliability prediction of a class in an object-oriented system. This step involves construction of a DAG for the BBN model based on the information about the fault model and the identified product metrics in the previous steps. Once

Operational Profile

System Reliability Prediction

System Reliability

Use Case Reliabilities

Phase 2: Use Case Reliability Prediction

Phase 3: System Reliability Prediction Legend Flow

Numerical Value

Functional blocks

123

272

the DAG is constructed the probability distribution is associated with every node of the constructed DAG. It can be observed in Algorithm 1 that procedure ESRA iterates on the use cases, the scenarios that realize each use case and the classes involved the execution of each scenario. In line number 8, the values of the object-oriented product metrics for each class are computed from UML diagrams. The computed product metrics and the number of times for which the class would be tested are assigned to the constructed BBN in line number 9 and 10 respectively. The reliability of each class is predicted from the constructed BBN model with help of BBN inference (Lauritzen and Spiegelhalter 1988; Paul and Michael 1993; Pearl 1988). In the next step (line number 13 of procedure ESRA), a matrix containing the EP probabilities among the classes is computed utilizing UML statechart specification and message passing information. The probability of EP is computed based on the method discussed in Sect. 2.2. The predicted reliabilities of classes involved in each scenario are modified based on the EP probability from the other classes. Based on these modified class reliability figures, the reliability of execution of a use case scenario is predicted in line number 15. In the next step, the execution


probability of each use case scenario is computed from operational profile, which in turn used to compute the reliability of a use case in line 18. The last step involves the reliability prediction of the overall system. For this, the probability of execution of each use case is computed from the operational profile in line 20. Subsequently, the predicted reliabilities of the use cases and the execution probabilities of the use cases are combined together to determine the reliability of the overall system in line 21. Finally, the predicted system reliability is produced as the output.

4 An illustrative example In this section, we illustrate the usage of our ESRA approach through a case study of a hotel automation system (HAS). The goal of such system is to automate the processes of the hotel, such as room booking, checking out, accounting activities and catering orders processing of the hotel. The system also generates statistical report about room tariff. Reception manger of the hotel can modify the hotel room rates according to the tariff. System also keeps

Algorithm 1 Pseudocode representing the steps of proposed ESRA approach

123


track of frequently visited customers for providing special discounts to them. The Catering service manager maintains prices all the items to be sold using this system. At the time of checking out, system generates bills for the customers. A use case model of the HAS application is shown in Fig. 7. It consists of five use cases, such as reserve room, register frequent guest, handle checkout, register food, revise room tariff and three actors: guest, reception manager, catering service manager. For the simplicity, we use U1, U2, U3, U4, U5 to denote the use cases. Each use case, except U1 and U3, is realized by one execution scenario. U1, U3 contains three and two execution scenarios respectively. In addition to the five use cases and the eight scenarios, HAS application consists of 13 classes. For the sake of readability we use Clj ðwhere; j ¼ 1; 2; . . .; 13Þ to denote the ith class and Sik to denote ith execution scenario of kth use case. The summary of characteristic of the HAS application has been shown in Table 4. The first, second and third column of the second row in Table 4 represent the total number of use case, total number of use case scenarios, and total number of classes present in HAS application respectively. The second, third, fourth, fifth and sixth column of the fourth row in Table 4 show the number scenarios involved in each use case. Again, the second, third, fourth, fifth and sixth column of fifth row contain the number of classes involved in all possible execution scenarios of each use case. 4.1 Class reliability prediction First, we predict the reliabilities of the classes. For this, we compute the object-oriented product metrics for each class of HAS application. We used our own custom-made

Reserve Room

Guest

Handle Checkout

Register Food Catering Service Manager

Revise Room Tarrif Hotel Automation Software

Fig. 7 An use case diagram of HAS

Table 4 Summary of the characteristic of HAS Number of use cases

Total number of scenarios

Total number of classes

5

8

13

In use case U1

In use case U2

In use case U3

In use case U4

In use case U5

Number of scenarios

3

1

2

1

1

Number of classes involved

4

3

3

4

3

metrics computation tool, DMetrics, to compute the values of the metrics UML diagrams of HAS. The detailed description of DMtrics is represented later in Sect. 5.2. The UML diagrams for the HAS application are drawn with the help of Magicdraw (http://www.magicdraw.com) and later they are exported into XML (extensible markup language) file from MagicDraw. DMetrics uses this XML file as an input and computes the object-oriented product metrics. Table 5 shows the summary statistic for the metrics set of HAS application. Once the design metrics for all the classes are calculated, these values are given as the inputs to our proposed BBN model for class reliability prediction. A commercial BBN modeling tool Netica (http://www.norsys.com/), is used to construct the proposed BBN model. The reliabilities of the classes of HAS application is estimated by utilizing Netica’s inference propagation algorithm (Markov chain Monte Carlo). Figure 8 shows a sample class reliability value obtained after entering the calculated design metrics values to the nodes labelled WMC, RFC, DIT, NOC, MIF, NOM, CIS, APM, DAM, NA, NS, NT in Netica. Table 6 shows the predicted reliabilities of the classes of HAS application with respect to the different number of test executions T. 4.2 Use case reliability prediction

Register Frequent Guest

Reception Manager

273

After predicting reliabilities of classes of HAS application, the reliabilities of the use cases are predicted. In order to perform this action, reliability of the each use case scenario are predict and later these predicted reliability figures are used to compute the reliabilities of use cases. First, use case U1 is selected for reliability prediction. U1 include three scenarios (S11, S21, S31) with execution probability of p1 = 0.8, p2 = 0.1 and p3 = 0.1 respectively. The execution probability of the scenarios within each use case is determined from the operational profile (Musa 1993). In the absence of operational profile information, it can be determined with help the of domain experts. In HAS application, objects of the classes Cl3, Cl4, Cl5, Cl6, Cl7

123

274


Table 5 A summary of HAS metrics statistic

Maximum

WMC

NOC

DIT

CBO

RFC

LCOM

CIS

DAM

MIF

NS

NT

14

18

19

11

12

22

15

6

5

0.49

0.37

Mean

2.32

1.22

1.03

2.26

1.27

1.43

1.79

0.31

0.29

1.31

1.72

Minimum

0

0

0

1

0

0

0

0

0

1

0

STDEV

1.63

1.04

1.59

2.35

1.78

0.31

2.43

0.56

0.21

5.26

3.12

scenarios S11 for use case U1 can be estimated using Eqs. 14 and 15 as follows: 0 1

participate in the execution of S11, S31 scenario and Cl3, Cl6, Cl7 in S21. The reliability of execution scenario of use case is computed based on the class reliability and the EP matrix which is determined by the methodology described in Sect. 2.2. The EP matrix is computed with the help of probability distribution of the set of state transition for each class, and probability distribution of the set of messages passed between each pair of class objects. To determine these probability figures, UML state diagram are used. The computed EP probability values are shown in Table 7. Each entry of Table 7 represents the EP probability from class present in row to class in the column. As an example, value of the cell (1, 3) in Table 7 represents the EP probability ðEð1; 3ÞÞ from class Cl1 to Cl3. Assuming the number of test execution T = 100 (each class will be tested 100 times), the reliability of the

RFC WMC 0 to 6 6 to 10 10 to 14

0 100 0 8 ± 1.2

0 to 4 4 to 8 8 to 12 12 to 16

Rw ðS11 Þ ¼

i¼3

6 ± 1.2

7 Y B C Rw ðCi ÞB 1 1 Rw ðCj Þ EðCj ; Ci ÞC @ A j¼3 j6¼i

¼ 0:836ð1 0:7321 0:865Þ 0:920ð1 ð0:3715 0:836Þð0:0021 0:865ÞÞ 0:865ð1 0:315 0:920Þ 0:895 0:956ð1 0:1863 0:865Þ ¼ 0:1246 where, Rw(S11) is reliability of the use case scenario S11. Like Rw(S11), reliability of the execution of use case scenarios S21 and S31 i.e. Rw(S21) and Rw(S31) is calculated

DAM 0 to 0.2 0.2 to 0.4 0.4 to 0.6 0.6 to 0.8

0 100 0 0

7 Y

NOC

0 100 0 0

0 to 3 3 to 6 6 to 9 9 to 12

0.3 ± 0.058

MIF

100 0 0 0

0 to 0.2 0.2 to 0.4 0.4 to 0.6 0.6 to 0.8

1.5 ± 0.87

100 0 0 0

0.1 ± 0.058

LCOM 0 to 8 8 to 16 16 to 24

DIT # Computational faults

100 0 0

0 to 2 2 to 4 4 to 6 6 to 8 8 to 10

4 ± 2.3 CBO 0 to 4 4 to 8 8 to 12

73.3 6.67 6.67 6.67 6.67 2.33 ± 2.6

# Initialization faults 0 to 2 2 to 4 4 to 6 6 to 8 8 to 10

CIS 100 0 0 0

73.3 6.67 6.67 6.67 6.67 2.33 ± 2.6

2 ± 1.2 # Logical faults

# Design faults 0 to 2 2 to 4 4 to 6 6 to 8 8 to 10

0 to 2 2 to 4 4 to 6 6 to 8 8 to 10

2.33 ± 2.6

100 0 0 2 ± 1.2

0 to 4 4 to 8 8 to 12 12 to 16

73.3 6.67 6.67 6.67 6.67

9.14 9.14 9.14 9.14 63.4 7.17 ± 2.8

0 to 2 2 to 4 4 to 6 6 to 8 8 to 10

6.67 11.3 61.6 13.8 6.67

# Inheritanc faults 0 to 2 2 to 4 4 to 6 6 to 8

78.6 7.14 7.14 7.14

NS 0 to 8 8 to 16 16 to 24 24 to 32

# State tran. faults

Number of faults 6.64 29.4 40.8 10.6 6.32 6.25

0 to 12 12 to 24 24 to 36 36 to 48 48 to 60 60 to 164.271

32.8 ± 25

78.6 7.14 7.14 7.14

Number of tests 100 0 0 0 0

Fig. 8 BBN model with values assigned to the independent nodes

0 100 0 0 12 ± 2.3 NT

1.86 ± 1.9

# Traceability faults 0 to 2 2 to 4 4 to 6 6 to 8

7.14 78.6 7.14 7.14 3.29 ± 1.4

Reliability of Class -0.0759768 to 0.2 0.2 to 0.4 0.4 to 0.6 0.6 to 0.8 0.8 to 1

0 to 8 8 to 16 16 to 24 24 to 32

100 0 0 0 4 ± 2.3

5.05 ± 1.9

50 to 100 100 to 200 200 to 400 400 to 800 800 to 1600

100 0 0 4 ± 2.3

1.86 ± 1.9

0 to 2 2 to 4 4 to 6 6 to 8

75 ± 14

123

0 to 8 8 to 16 16 to 24

# Contl. & seq. faults

7.08 6.67 9.58 18.2 58.4

0.726 ± 0.26


275

way as U1. Predicted reliabilities of use cases of the system and usage frequencies of the use cases are shown in Tables 8 and 9 respectively.

Table 6 Predicted classes reliabilities Class

Reliability T = 50

T = 100

T = 500

Cl1

0.870

0.935

0.987

Cl2

0.972

0.986

0.997

Cl3

0.672

0.836

0.967

Cl4

0.840

0.920

0.984

Cl5

0.730

0.865

0.973

Cl6

0.790

0.895

0.979

Cl7 Cl8

0.912 0.750

0.956 0.875

0.991 0.975

Cl9

0.670

0.835

0.967

Cl10

0.882

0.941

0.988

Cl11

0.598

0.799

0.959

Cl12

0.758

0.879

0.975

Cl13

0.826

0.913

0.982

4.3 System reliability prediction Reliability of a system can be predicted using the reliabilities of use cases of the system and execution probability of the use cases with the help of Eq. 17. The probability of execution of each use case is determined from the operational profile (Musa 1993). The probability of operational usage of five uses are given in Table 9. The reliability of the HAS is estimated as follows: Rw ðSysÞ ¼

5 Y

ð1 ð1 Rw ðUi ÞÞ Pi Þ

i¼1

¼ ð1 ð1 0:99630Þ 0:18Þ ð1 ð1 0:99876Þ 0:11Þ ð1 ð1 0:88107Þ 0:24Þ

and their values are 0.357 and 0.1935 respectively. Using Eq. 16 reliability of U1 is estimated from the reliabilities of use case scenarios (S11, S21, S31) and execution probability of these scenarios (p1 = 0.8 p2 = 0.1 and p3 = 0.1 respectively) as follows: Rw ðU1 Þ ¼ 1

3 Y

ð1 ð1 0:97328Þ 0:32Þ ð1 ð1 0:89351Þ 0:15Þ ¼ ð1 0:000666Þ ð1 0:000136Þ ð1 0:028543Þ ð1 0:008550Þ ð1 0:015973Þ ¼ 0:947006

ðð1 Rw ðSi1 ÞÞ pi Þ

i¼1

The predicted reliability value of the HAS application can change with the change of number of test case executions T. The change of reliability values of HAS application with respect to the number test executions has been plotted in Fig. 9. It should also be noted that the predicted reliability value is the worst case reliability of the application with respect to the number of tests to be performed over it.

¼ 1 ½ðð1 0:1246Þ 0:8Þ ðð1 0:357Þ 0:1Þ ðð1 0:1935Þ 0:1Þ ¼ 1 0:003631 ¼ 0:996368 Reliabilities of the other four use cases (U2, U3, U4, U5) that are used to model the system, are estimated in a similar Table 7 EP among the classes Cl1

Cl2

Cl3

Cl4

Cl5

Cl6

Cl7

Cl8

Cl9

Cl10

Cl11

Cl12

Cl13

Cl1

0

0

0.0323

0

0

0

0

0

0

0

0

0

0

Cl2

0.6213

0

0

0

0

0

0.0143

0

0

0

0

0

0

Cl3

0

0

0

0.3715

0

0

0

0

0.245

0

0.021

0

0

Cl4

0

0

0

0

0.315

0

0

0

0

0

0

0

0

Cl5

0

0

0.7321

0.0021

0

0

0.1863

0

0

0

0

0

0

Cl6

0

0

0

0

0

0

0

0.5247

0

0

0

0

0.00281

Cl7

0.3821

0

0

0

0

0

0

0

0

0.0378

0

0.237

0

Cl8 Cl9

0 0

0 0.272

0 0

0 0

0 0

0 0

0 0.4821

0 0

0 0

0.013 0

0.721 0

0 0

0 0

Cl10

0

0

0

0

0

0

0

0.0891

0

0

0

0.0421

0

Cl11

0

0

0.0985

0

0

0.523

0

0

0.0786

0

0

0

0

Cl12

0

0

0

0

0

0

0.0724

0

0

0

0

0

0.632

Cl13

0.3751

0

0

0

0

0.0981

0

0

0.278

0

0

0

0

123

276


Table 8 Predicted use case reliabilities Use case

Reliability T = 50

T = 100

T = 500

U1

0.9816

0.9963

0.9974

U2

0.9968

0.99876

0.9993

U3

0.8673

0.88107

0.9286

U4

0.9133

0.97328

0.9865

U5

0.8216

0.89351

0.9352

Kharagpur. All participating students who had the previous experience of preparing UML models and programming in Java language, were divided into ten groups which consist of five members for each. Each group was assigned with a project to develop a software application that models a real world problem. The participants developed the software applications using similar software engineering process and implementation environment to eliminate any variations to reliabilities due to disparate processes and implementation environments. 5.2 Software tools used

Table 9 Probabilities of executions of the use cases Use case

U1

U2

U3

U4

U5

Usage frequency

0.18

0.11

0.24

0.32

0.15

A few number of tools have been used in our experimental studies. In the following we briefly mention these tools. –

–

–

Fig. 9 Predicted reliabilities of HAS application by ESRA approach

5 Experimental studies In this section, we present a set of experiments that were performed to empirically evaluate our ESRA approach. In the following, we first briefly discuss about the experimental setup and data used in our case study. Subsequently, we discuss the software tools that have been used to perform the experimental studies. Next, we present the set of experiments that have been performed for empirical evaluation of our ESRA approach. Finally, we analyze the results of our experimental studies. 5.1 Experimental data and setup In the absence of any benchmark or industry data, our experimental study is based on data collected from student projects. These projects were carried out by the undergraduate students of department of computer engineering of different engineering colleges of eastern India and IIT

123

UML Modeling Tool: A commercial tool, namely (Magicdraw. http://www.magicdraw.com) has been used for construction UML models that have used in our study. MagicDraw is selected for our experimental studies, as it has easy to graphical use interface (GUI) and it can export the UML diagrams in XML files. BBN Model Construction Tool: The proposed BBN model for class reliability prediction has been constructed using a commercial tool (Netica. http://www.norsys.com/ ). Netica is a GUI-based BBN modeling tool which uses Markov Chain Monte Carlo (MCMC) algorithm for inference propagation. Metrics Computation Tool: A large number of commercial tools are available for computing a subset of object-oriented product metrics that are used for this study. However, no commercial tool is currently available to support computation of all metrics together. Moreover, most of these tools compute the values of the product metrics based on the source code of the software. Therefore, we developed a GUI based tool, DMetrics, to compute various object-oriented product metrics from UML diagrams. DMetrics has been developed using Java programming language and can run on multi-platform environment. A schematic and a snapshot of DMetrics is represented in Fig. 10a and b respectively. In the following we briefly mention the implementation of DMetrics. As shown in Fig. 10a, DMetrics takes as input a UML architectural model comprising of the class, use case, sequence and state diagrams in XML format which can be generated by MagicDraw. This XML file is parsed by the Parser Module, to extract information regarding different classes, their attributes and operations from the structural model. This module also gathers behavioral information regarding objects of different classes by analyzing the state and sequence diagrams. The information obtained using the Parser Module is then


(a)

277

(b)

Fig. 10 A representation of DMetrics tool for design metrics computation. a A schematic of DMetrics, b a snapshot of DMetrics

used to initialize all the data structures needed to compute the relevant design metrics. Analyzing the structural and behavioral information stored in the data structures, Metrics Computation Module computes the relevant object-oriented design metrics based on the metrics option selected by the user. DMetrics supports a GUI for all user interactions such as specification of metrics types, display of metrics values in tabular format, export metrics in comma-separated values (csv) file, etc.

5.3 Experimental tasks and objectives Three set of tasks were performed with their respective objectives. First, we have applied ESRA to predict the reliability of each software application from its objectoriented product metrics which were extracted from prepared UML models by each student group. In order to perform this task, the groups were asked to prepare UML models based on the analysis of requirement specifications of the projects assigned to them. Subsequently, the UML models which consisted of a few class diagrams, sequence diagrams and state diagrams, were exported into XML files. The XML files were used as the inputs to our DMetrics tool to compute the object-oriented product metrics. Among all the design metrics, the relevant ones which have been identified in Sect. 3.1.2. were selected and assigned to the BBN model for class reliability prediction. Subsequently, the reliabilities of the use cases were predicted based on the class reliabilities predicted from the BBN model and the operation profile information. Finally, reliability of each software application is predicted

utilizing the reliabilities of its use case and operational profile. The detailed steps which were followed to predict reliability of each system from object-oriented design metrics are discussed in Sect. 4. Second task involves estimation of actual reliability or the observed reliability of each software application through testing of source code of the application. This task was performed to compare reliability figures predicted in the previous task with the observed reliability figures. In effects of this process, each student group was asked to develop the source codes of the assigned problems using Java programming language. The Java source codes were developed conforming the UML models which were used by ESRA approach to predict reliability. Once the coding of the applications were finished, the five teams consisting a few master students and the experienced stuff members of Department of Computer Science at IIT Kharagpur, carried out testing of the developed applications. Each application was executed with the test cases chosen randomly from the input domain of the application constrained by its operational profile. After each run with a test case, the total failure count which was set zero at the beginning of test, was increased by one if any failure was observed. The reliability of the student software project is estimated with the help of the following equation: nf Ro ¼ 1 ð18Þ T where Ro is the observed reliability of the application, nf is total number of failures observed in the testing process, T is the total number of test executions. Due to space limitations, we do not include details of the testing process here. The third task was carried out to evaluate the performance of our ESRA approach with respect to an

123

278

established approach for reliability prediction. For this task, we have chosen the approach proposed by Rome Air Forces Development Centre (RADC) (McCall et al. 1992) for performance comparison. We have chosen this approach (McCall et al. 1992) since this work is frequently referred in the literature and more related to our approach. Moreover, the assumptions made by RADC approach are more the realistic compared to other reliability predict approaches (Cheung et al. 2008; Gokhale and Trivedi 2002; Yacoub et al. 1999). A few criteria have been used to compare the performance of the proposed ESRA approach and RADC approach. The first one was root mean square prediction error (RMSPE) which is the measure of how close the predicted values are to observed values. RMSPE is defined in the following equation. pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi RMPSE ¼ variance2 þ bias2 ð19Þ where Bias and Variance are computed using the following equations. sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi n 1 X ð20Þ ðmi mî biasÞ2 Variance ¼ n 1 i¼1 where n is the number of experiments, mî is the predicted value and mi is the observed value. Bias is computed as follows: Pn ðmî mi Þ ð21Þ Bias ¼ i¼1 n Mean absolute error (MAE) was considered as the second criteria for performance comparison. MAE is the measure of average deviation percentage in prediction. The MAE value closer to zero is considered as the better prediction capability by the model. MAE is defined as follows: n 1X MAE ¼ jmî mi j ð22Þ n i¼1 where n is the number of experiments, mî is the predicted value and mi is the observed value. 5.4 Results and discussion The results obtained from our experiments have been summarized in Tables 10 and 11. The first column in Table 10 shows the type of programs that are developed by the student teams. Column 2 shows the number of predicted reliabilities of the corresponding system by our ESRA approach. Column 3 represents the actual or the observed reliabilities for each corresponding system developed by the student groups. In column 4, we present the predicted reliabilities of the software applications by RADC approach (McCall et al. 1992). It should be noted

123


that the reliability figures that are presented in Table 10 has been obtained after testing each application for 100 number of times. Table 11 shows the results of our performance comparison. Column 1 of this table represents the name of the comparison criteria. In columns 2 and 3 of Table 11, we present the values obtained using ESRA and RADC approaches with respect to each comparison criteria respectively. It can be observed from Table 11 that ESRA has lower prediction error than that of RADC approach, as RMPSE value for ESRA approach is lower than to RADC approach. Moreover, from Table 11, we can observe that the accuracy of prediction (presented in terms of MAE) achieved by ESRA is higher than that achieved by RADC approach. The reliability predicted by ESRA approach, RADC approach (McCall et al. 1992) and the observed reliability of each software systems are presented in form of a bar graph in Fig. 11.

6 Comparison with related work Although various issues associated with software reliability (Cheung et al. 2008; Cortellessa et al. 2002; Gokhale and Trivedi 2002; McCall et al. 1992; Yacoub et al. 1999) have been widely studied in literature, little work has been done to address the problem of reliability prediction of objectoriented applications during early stages of product development. Singh et al. have proposed a reliability prediction model for component-based systems (Cortellessa et al. 2002). They used a Bayesian approach to compute the model parameters from prior information about the failure probabilities of the components and system usage. Later, they used this model to predict the reliability of system during design phase with the help of properly annotated use case diagrams and sequence diagrams. Hence, this is considered as reliability prediction approach for object-oriented software application. However, their approach does not consider the reliability of the interfaces among the components, which may lead to overestimation of the system reliability. The reliability predictive model proposed by the RADC (Cortellessa et al. 2002) quantifies the reliability of a system in terms of its failure rate. At first, fault density of a software application is estimated using software engineering measures. Subsequently, this fault density is converted to failure rate using certain conversion ratios (McCall et al. 1992) which were proposed by the RADC. According to this model, fault density of a system determined by the product of application type, development environment, and the values of a set of software metrics (S). The set of software metrics (S) consists of product and process metrics that can be determined in different phases of software


279

Table 10 Summary of the predicted reliabilities by ESRA and RADC approach and observed reliabilities Name of the system

ESRA approach

Observed reliability

RADC approach

Hotel Automation System (HAS)

0.947006

0.97

0.8752

Library Information System (LIS)

0.894023

0.95

0.8371

Bookshop Automation System (BAS)

0.893211

0.96

0.8232

Supermarket Automation Software (SAS)

0.762102

0.81

0.8821

Judiciary Information System (JIS)

0.843315

0.92

0.7241

Medical shop Automation Software (MAS)

0.745861

0.80

0.7003

Time Management Software (TMS)

0.812441

0.89

0.7001

Road Repair and Tracking System (RRTS)

0.713823

0.79

0.8506

Word Processing Software (WPS) Software for Hall Management Centre (SHMC)

0.898302 0.842693

0.91 0.92

0.7113 0.9521

Table 11 A performance comparison of ESRA approach with respect to RADC approach

RMPSE MAE

ESRA approach

RADC approach

0.0614 0.05672

0.1361 0.11935

development. This approach is as sound and similar as of ours, in predicting reliability from the design metrics perspective. However, it does not consider the object oriented metrics and treat the software as a monolithic element. Hence, the use of this approach in reliability prediction of software components of the system is not possible. Cheung et al. (10) proposed a framework in which they first predict component reliability. They incorporate a failure state in the UML state models for individual components and apply a discrete time Markov chain stochastic process to those models. They determine the reliabilities of individual components from the probability of the component being in a normal state (not in the failure state) by applying standard numerical techniques to solve the Markov chain model. System reliability is determined from the reliabilities of its components. Practical use of their approach is restricted as accurate determination of the state-transition probabilities of various components is difficult. Consequently, when the number of components of the system increases, this approach may suffer from stateexplosion problem. Moreover, this approach assumes that the transfer of control among the components follows firstorder Markov property which may not hold for many software applications, where the execution history determines which component is the next one to be executed. Compared to the above approaches, ESRA is a bottomup approach, in which reliabilities of classes in an objectoriented system are predicted first. For this process, the main focus is given to predict the major categories of faults that can occur in a class instead of predicting faults as a

Fig. 11 Predicted reliabilities by ESRA and RADC approach and observed reliabilities

123

280

whole. The expected values of different types of faults that are estimated based on the object oriented product metrics correlated to these faults types. Subsequently, The estimated number of faults present in a class and number of test that will be performed over the class is used to determine the reliability of the class. Based on the reliabilities of classes and operational profile, reliabilities of the use cases are predicted. Moreover, in use case reliability prediction, the amount of EP among the different classes are taken into account. In other words, the proposed approach considers the failure dependency among the components of the system. Finally, the system reliability is predicted based on the reliability and the execution probability of the use cases.

7 Conclusion In this paper, we propose a new bottom-up approach to predict the reliability of object-oriented systems during the early stages of the product development. Our proposed approach first predicts the reliabilities of the classes in an object-oriented system using product metrics extracted from the UML specifications of a software system. The reliability of the overall system is predicted based on operation profile and reliabilities of classes and UML artifacts. Some of the salient features of our proposed approach include the following: In addition to overall system reliability, we predict the reliabilities of the classes, scenarios and use cases, which helps to identify the less reliable classes, scenarios and use cases of the system. Moreover our proposed approach uses a Bayesian network and its underlying forward inference propagation algorithm to compute the reliability from the design metrics. Henceforth the same Bayesian network can be utilized by applying backward inference propagation algorithm to determine the possible values of the product metrics for a given reliability target. In other words, our proposed approach helps to select the suitable design among the available different design choices which leads to the achievement of the required level of reliability for the overall system. Analysis of the results obtained from our experimental studies shows that our approach produces approximately 6.26% lower MRE and 7.47% lower RMPSE compared to RADC approach (McCall et al. 1992). The possible extension of our work would be to explore the relationship among various process metrics and different types of faults that can occur in a software product. Simultaneously, we plan to extend our ESRA approach by incorporating those process metrics which will show strong correlation with different types of faults. We also intend to apply our approach to the real-life systems to investigate the accuracy of the prediction.

123


References Abdelmoez W, Nassar DM, Shereshevsky M, Gradetsky N, Gunnalan R, Ammar HH, Yu B, Mili A (2004) Error propagation in software architectures. In: METRICS ’04: proceedings of 10th international symposium on the software metrics. IEEE Computer Society, Washington, DC, pp 384–393 Abreu FBE, Henderson-Sellers B, Piattini M, Poels G, Sahraoui HA (2002) Quantitative approaches in object-oriented software engineering. In: ECOOP ’01: proceedings of the workshops on object-oriented technology. London, UK, Springer-Verlag, pp 174–183 Bai CG, Hu QP, Xie M, Ng SH (2005) Software failure prediction based on a Markov Bayesian network model. J Syst Softw 74(3):275–282 Bansiya J, Davis CG (2002) A hierarchical model for object-oriented design quality assessment. IEEE Trans Softw Eng 28(1):4–17 Binder RV (1996) Testing object-oriented software: a survey. Softw Test Verif Reliab 6(3–4):125–252 Bishop PG (2002) Rescaling reliability bounds for a new operational profile. In: ISSTA ’02: proceedings of the 2002 ACM SIGSOFT international symposium on software testing and analysis. ACM, New York, pp 180–190 Cai KY, Hu DB, Bai CG, Hu H, Jing T (2008) Does software reliability growth behavior follow a non-homogeneous poisson process. Inf Softw Technol 50(12):1232–1247 Cheung L, Roshandel R, Medvidovic N, Golubchik L (2008) Early prediction of software component reliability. In: ICSE ’08: proceedings of the 30th international conference on software engineering. ACM, New York, pp 111–120 Chidamber SR, Kemerer CF (1994) A metrics suite for object oriented design. IEEE Trans Softw Eng 20(6):476–493 Cortellessa V, Grassi V (2007) A modeling approach to analyze the impact of error propagation on reliability of component-based systems. In: CBSE ’07: proceedings of the 10th international conference on component-based software engineering. Springer, Berlin, pp 140–156 Cortellessa V, Singh H, Cukic B (2002) Early reliability assessment of UML based software models. In: WOSP ’02: proceedings of the 3rd international workshop on Software and performance. ACM, New York, NY, USA, pp. 302–309 Dagum P, Chavez RM (1993) Approximating probabilistic inference in Bayesian belief networks. IEEE Trans Pattern Anal Mach Intell 15:246–255 Fenton NE, Neil M (1999) A critique of software defect prediction models. IEEE Trans Softw Eng 25(5):675–689 Genero M, Miranda D, Piattini M (2002) Defining and validating metrics for UML statechart diagrams. In: QAOOSE ’02: 6th international ECOOP workshop on quantitative approaches in object-oriented software engineering, pp 120–136 Gokhale SS, Trivedi KS (2002) Reliability prediction and sensitivity analysis based on software architecture. In: ISSRE ’02: proceedings of the 13th international symposium on software reliability engineering. IEEE Computer Society, Washington, DC, pp 64–75 Huang CY (2005) Performance analysis of software reliability growth models with testing-effort and change-point. J Syst Softw 76(2):181–194 Inoue S, Yamada S (2010) Integrable difference equations for software reliability assessment and their applications. Int J Syst Assur Eng Manag 1(1):5–10 Lanza M, Marinescu R, Ducasse S (2005) Object-oriented metrics in practice. Springer-Verlag New York, Inc., Secaucus Lauritzen SL, Spiegelhalter DJ (1988) Local computations with probabilities on graphical structures and their application to expert systems. J R Stat Soc B (Methodol) 50(2):157–224

Int J Syst Assur Eng Manag (Oct-Dec 2011) 2(4):261–281 Magicdraw. Available at http://www.magicdraw.com/ Malhotra R, Kaur A, Singh Y (2010) Empirical validation of objectoriented metrics for predicting fault proneness at different severity levels using support vector machines. Int J Syst Assur Eng Manag 1(3):269–281 McCall JA, Randell W, Dunham J, Lauterbach L (1992) Software reliability, measurement, and testing software reliability and test integration. Tech. rep. Final Technical Report RL-TR-92-52, Rome Laboratory, Rome Mohamed A, Zulkernine M (2008) On failure propagation in component-based software systems. In: QSIC ’08: proceedings of the 2008 the eighth international conference on quality software. IEEE Computer Society, Washington, DC, pp 402–411 Mohanta S, Vinod G, Ghosh AK, Mall R (2010) An approach for early prediction of software reliability. ACM SIGSOFT Softw Eng Notes 35(6):1–9 Musa J (1993) Operational profiles in software-reliability engineering. IEEE Softw 10(2):14–32 Netica. Available at http://www.norsys.com/ Offutt J, Alexander R, Wu Y, Xiao Q, Hutchinson C (2001) A fault model for subtype inheritance and polymorphism. In: ISSRE ’01: proceedings of the 12th international symposium on software reliability engineering. IEEE Computer Society, Washington, DC, pp 84–93 Olague HM, Etzkorn LH, Gholston S, Quattlebaum S (2007) Empirical validation of three software metrics suites to predict

281 fault-proneness of object-oriented classes developed using highly iterative or agile software development processes. IEEE Trans Softw Eng 33(6):402–419 Paul D, Michael L (1993) Approximating probabilistic inference in bayesian belief networks is NP-hard. Artif Intell 60(1):141–153 Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann Publishers, San Francisco Reussner RH, Schmidt HW, Poernomo IH (2003) Reliability prediction for component-based software architectures. J Syst Softw 66(3):241–252 Tripathi R, Mall R (2005) Early stage software reliability and design assessment. In: APSEC ’05: proceedings of the 12th Asia-Pacific software engineering conference. IEEE Computer Society, Washington, DC, pp 619–628 Yacoub SM, Cukic B, Ammar HH (1999) Scenario-based reliability analysis of component-based software. In: ISSRE ’99: proceedings of the 10th international symposium on software reliability engineering. IEEE Computer Society, Washington, DC, pp 22–31 Yuan C, Druzdzel MJ (2007) Generalized evidence pre-propagated importance sampling for hybrid Bayesian networks. In: Proceedings of the 22nd national conference on artificial intelligence—vol 2. AAAI Press, Vancouver, pp 1296–1302

123

A technique for early prediction of software reliability ... - Springer Link

A technique for early prediction of software reliability ... - Springer Link

Suggest Documents

Technique for Early Reliability Prediction of Software ... - PLOS

Technique for Early Reliability Prediction of ... - Semantic Scholar

An approach for early prediction of software reliability

Towards early software reliability prediction for computer forensic tools ...

Towards early software reliability prediction for computer forensic tools ...

Encapsulation of legacy software: A technique for ... - Springer Link

A Retrieval Technique for Software Components Using ... - Springer Link

A Retrieval Technique for Software Components Using ... - Springer Link

Application of sliding window technique for prediction ... - Springer Link

Parameterized Reliability Prediction for Component-Based Software ...

Parameterized Reliability Prediction for Component-Based Software ...

Reliability prediction for component-based software architectures

Early Reliability Prediction: An Approach to Software ... - CiteSeerX

Early Reliability Prediction: An Approach to Software ... - CiteSeerX

Early Quantitative Software Reliability Prediction Using Petri-nets

Software Reliability Early Prediction in Architectural Design Phase ...

a method for the prediction of software reliability - CiteSeerX

a method for the prediction of software reliability - CiteSeerX

Early Adolescent Depressive Symptoms: Prediction ... - Springer Link

Enhancing Accuracy of Software Reliability Prediction

Reliability and efficacy of the new massage technique ... - Springer Link

A heuristic approach for software reliability prediction - CUHK CSE

Bilateral Rib-to-pelvis Technique for Managing Early ... - Springer Link

Using a Reliability Growth Model to Control Software ... - Springer Link