A Rigorous Method for Inspection of Model-Based ... - Semantic Scholar

IEEE TRANSACTIONS ON RELIABILITY, VOL. 59, NO. 4, DECEMBER 2010

667

A Rigorous Method for Inspection of Model-Based Formal Specifications Shaoying Liu, Senior Member, IEEE, John A. McDermid, and Yuting Chen

Abstract—Writing formal specifications can help developers understand users’ requirements, and build a solid foundation for implementation. But like other activities in software development, it is error-prone, especially for large-scale systems. In practice, effective detection of specification errors still remains a challenge. In this paper, we put forward a rigorous, systematic method for the inspection of model-based formal specifications. The method makes good use of the well-defined consistency properties of a specification to provide precise rules and guidelines for inspection. The inspection process utilizes both well-defined expressions derived from the specification and human inspectors’ judgments to find errors. We present a case study of the method by describing how it is applied to inspect an Automated Teller Machine (ATM) software specification to investigate the method’s feasibility, and explore potential challenges in using it. We also describe a prototype software tool including its functions and distinct features to demonstrate the tool supportability of the method.

NOTATION Denotes a process specification, where , , and are the sets of input data flow variable declarations, output data flow variable declarations, and external variable declarations, respectively; , and denote the preand post-conditions of process , respectively. Denotes the element set of the sequence . represents the length of the sequence . refers to the element of the sequence at the position indicated by the .

Index Terms—Formal analysis, formal specification, rigorous inspection, verification.

yields a concatenation of two sequences and .

ACRONYM

means that the pair of input , and the of external variable initial value has the relation with the pair of output , and the final value of external variable .

ITT

Inspection Task Tree

RIM

Rigorous Inspection Method

ATM

Automated Teller Machine

SOFL

Structured Object-oriented Formal Language

DFD

Data Flow Diagram

CDFD

Condition Data Flow Diagram

VDM

Vienna Development Method

FS

Functional Scenario

ADR

Active Design Review

I. INTRODUCTION

M

Manuscript received December 01, 2008; revised March 01, 2010; accepted May 24, 2010. Date of publication November 09, 2010; date of current version November 30, 2010. This work is supported in part by the Ministry of Education, Culture, Sports, Science, and Technology of Japan under a Grant-in-Aid for Scientific Research on Priority Areas (16016279) and on Foundation C (18500027). S. Liu is also supported by the NSFC Grant 60910004 and 973 Program of China Grant 2010CB328102. Associate Editor: C. Smidts. S. Liu is with the Department of Computer Science, Faculty of Computer and Information Sciences, Hosei University, Koganei, Japan. J. A. McDermid is with the Department of Computer Science, University of York, York, UK. Y. Chen is with the School of Software, Shanghai Jiaotong University, Shanghai, China. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TR.2010.2085571

ODEL-BASED formal specification has been recognized as an effective means for helping developers to understand users’ requirements by precisely defining functions, data resources, and necessary constraints in a well-organized structure [1]–[3]. In general, formal specifications need to be written by humans, and the process is unlikely to be automatic. Therefore, the resulting specifications are highly likely to contain errors; and eliminating the errors in the early phase of a software project can produce a considerable positive effect on the overall cost of the project, and the reliability of the final software product [4]. The errors may be made because of inaccurate understanding of the requirements, incorrect uses of mathematical expressions, or wrong decisions in defining data or functions (e.g., choosing wrong data structures or wrong operators). To ensure that a satisfactory program is developed on the basis of a specification, errors must be detected and removed from the specification through verification. The question is how to carry out a verification so that the verification activity can be made rigorous, systematic, and practical for software projects using formal specifications.

0018-9529/$26.00 © 2010 IEEE

668

Several techniques are available for specification verification, such as formal proof [5], model checking [6], animation [7], and conventional inspection [8]. But unfortunately they all face serious challenges in practice. Formal proof provides a fundamental technique that can be used to verify consistency properties of specifications, but its cost-effectiveness and practicality are controversial [9], [10]. According to our experience, some consistency properties, such as feasibility of operations, are extremely difficult to prove. And the proof process is complex, and in general cannot be fully automated. The failure of a proof, which may result from the existence of errors in the related properties or the use of inappropriate proof tactics, inference rules, or hypotheses, requires a human to identify and fix the problem. But there is no specific guideline available for this procedure. Model checking is another possible technique for verification of software artifacts (e.g., designs) [11], and it has the advantage that the verification can be performed automatically. However, it suffers from the state explosion problem. It is also difficult to apply to data-intensive applications because the treatment of the data usually results in an infinite state space [12], [13]. The state explosion problem may be alleviated by abstraction in functions, data types, or both. But the abstraction requires high-level skills, and its correctness may not be guaranteed. Unlike formal proof and model checking, specification animation was developed as a technique for verifying the validity of specifications. As Miller and Strooper point out [7], specification animation provides end users and field experts with a chance to interact with the specification, and observe its operational behavior, so that they can check whether the specification reflects the intended properties of their design. Some tools have been built to support animation of different specification languages, such as PiZA [14], and B-Model animator [15]. But most of those tools follow an approach similar to testing, namely executing specifications with sample input values, and analyzing the results. In general, this kind of technique requires a translation from a formal specification language to an executable programming language (e.g., Prolog or LISP), which imposes many restrictions on the style of the specification. It also faces the same challenge as testing in selecting enough adequate “animation cases”. Conventional inspection is a static analysis technique for detecting errors in software products (e.g., programs) [16], [17]. According to the IEEE standard [18], the purpose of inspection is to detect and identify software product anomalies, including software errors (or defects) and deviations from standards and requirements. Inspection is usually based upon a checklist that contains a collection of questions concerning the qualities and standards of software. Inspection can also be applied to high-level documentation, such as software specification and design [19], [20]. In comparison with formal proof and model checking, inspection emphasizes the deployment of inspectors’ skill and judgment in finding errors, and is therefore applicable to both formal, and informal specifications. Unlike the animation technique, inspection does not require the specification to be executable, or to follow a certain style; it can be applied to both un-executable, and executable specifications. In spite of the advantages of inspection over the other three techniques, the effectiveness of its application however relies


heavily on the skills and experience of inspectors, and the focus of inspection tools is almost entirely on administrative tasks such as scheduling meetings, and collecting defect reports [8], [20]. The primary reason for this situation is the lack of formality in the artifacts (e.g., specifications) being inspected, the lack of formality in the questions on checklists, and the lack of formality in the inspection discipline. Without sufficient formality, inspection targets cannot be precisely expressed, and the inspection process cannot be supported in depth by software tools. When inspecting formal specifications, however, there is an opportunity to exploit the formalism in specifications to improve the rigor and tool supportability of specification inspection. In general, the inspection of a formal specification can be divided into two levels. The first level is to inspect whether all of the informal functional requirements and data items are represented in the formal specification. Because of the informality of the requirements, the inspection is usually difficult to tell whether the formal specification satisfies the requirements semantically. It can only concentrate on the traceability, checking whether every functional requirement is represented by a set of functional scenarios (see Section II-A for a detailed explanation of this concept), and whether every required data item is reflected in the formal specification. Such an inspection ensures that sufficient functional scenarios and data structures are used in the formal specification to represent all of the informal requirements. The second level inspection can be adopted to check the formal specification semantically, ensuring that it is internally consistent in terms of satisfying necessary properties, and valid in terms of capturing the true needs of the user. Whereas the first level inspection has been studied in our previous work on integrating formal specification, review, and testing [21], the second level inspection has not been well researched. This paper describes a novel inspection method for the second level inspection, known as the rigorous inspection method (RIM). RIM features preciseness, visualization, and simplicity through three combinations. First, it combines formal analysis and a visual notation in the way formal analysis is adopted to derive functional scenarios, form precise inspection tasks (statements of required checks of specification properties) on the checklist, and provide rigorous guidance for inspecting the specification, while the visual notation called Inspection Task Tree (ITT) is used to provide an intuitive visual representation of each inspection task. Second, it combines the strength of mathematics used in formal proof with the human inspector’s role. Mathematics serves as a means of precisely expressing specification properties involved in the tasks on the checklist, while the inspector analyses the properties and the related statements of the specification, and makes decisions on errors. Third, it combines the verification of consistency properties into the validation of specification statements to achieve the effect of “killing two birds with one stone”. In general, formalizing informal requirements into a formal specification requires further decisions, for example, in interpreting the requirements in detail, and in choosing appropriate data structures and mathematical expressions. The formal specification obtained through such a formalization is very difficult to be validated by only checking it against the informal requirements

LIU et al.: A RIGOROUS METHOD FOR INSPECTION OF MODEL-BASED FORMAL SPECIFICATIONS

specification because many of the decisions reflected in the formal specification are made during the formalization process. Inspection of the decisions with the guidance derived from consistency properties offers a practical solution. We believe that the features above allow RIM to strike a better balance between formality and practicality than the other approaches mentioned previously. To find out the feasibility of our method, we have conducted a case study by applying it to the inspection of an Automated Teller Machine (ATM) software specification [22]. The case study is intended to allow us to learn about the performance of the method, find potential challenges in using it, and investigate how the method can be well supported by a software tool (see Section V for details.) Benefiting from the experience of this case study, we have also developed a prototype software tool to demonstrate the supportability of our method. The tool offers assistance to the inspector by automatically deriving inspection tasks from the specification, constructing an ITT for each task, guiding the inspector through the inspection process, and evaluating the inspection results (see Section VI for details.) Our major contributions in this paper are twofold. One is the establishment of the rigorous inspection method RIM, including its underlying principle, representation notation, and inspection procedure. The method’s originality lies in its capability of bridging formal verification and practical inspection techniques, allowing inspections of formal specifications to be performed intuitively on a solid theoretical foundation. Another main contribution is the development of a prototype software tool for RIM. Its value lies in the demonstration of the tool supportability of RIM, which is necessary and important for technology transfer from research to practice. The remainder of the paper is organized as follows. Section II describes the principle of RIM, while Section III discusses the consistency properties for forming inspection tasks. In Section IV, we discuss the inspection process based upon the graphical notation ITT, including the construction of trees to present inspection tasks, reading techniques, and inspection evaluation. Section V describes a case study of RIM. Section VI presents a prototype software tool for RIM. In Section VII, we describe related work, and compare it with RIM. Finally, in Section VIII, we conclude the paper, and point out future research directions. II. PRINCIPLE OF INSPECTION We describe the principle of RIM by discussing the inspection strategy, and the inspection procedure. Our discussions in this section and throughout the entire paper are based upon an assumption that the formal specification to be inspected has passed the first level inspection mentioned in the Introduction section, ensuring that it contains functional scenarios, data types, and variables that are sufficient to represent all of the required functions and data items in the informal requirements document. We also use the Structured Object-oriented Formal Language (SOFL) as the formal specification language in the discussions [21]. SOFL results from integration of conventional data flow diagrams (DFDs) [23], the Vienna Development Method (VDM) [24], and Petri Nets [25] to achieve preciseness, visualization, and simplicity in a practical modeling

669

technique. The DFD notation is extended to a formal graphical notation called Condition Data Flow Diagram (CDFD) that has a set of well-defined operational semantics similar to that of a Petri Net [26]. A process in a CDFD is treated as a transition, and a data flow as a token; and when all of the input data flows of the process become available, the process will be enabled and executed. The CDFD is a formal but intuitive notation for describing system architecture, while the VDM-SL (VDM-Specification Language) is adopted to define the components of the CDFD, such as processes, data flows, and data stores. SOFL was developed through extensive research [27], [28], and has been learned by thousands of university students, researchers, and practitioners. It has also been applied in several industrial and academic projects [29]–[31], and is currently being applied by researchers to develop financial systems, transport information systems, and enterprise information systems. We choose SOFL for the discussion of our inspection method in this paper partly because of our expertise and its potential impact on industrial software development, and partly because SOFL shares many common features with some well-known formal notations (e.g., VDM [24], B-Method [32]). For this reason, the underlying principle of the inspection method described in this paper can also be applied to other model-based formal notations (e.g., VDM, UML OCL [33]). A. Inspection Strategy As discussed previously, the overall goal of specification inspection is to achieve a consistent, valid specification by detecting and eliminating errors. But the important question is how to carry out the inspection so that we can reach the goal effectively. Let be a specification defining an abstract model of a system under development. If is consistent and valid, then for any desired input , defines a desired output . By desired input or output, we mean that it satisfies the designer’s intention (the person who writes the specification is called a designer). To ensure the consistency and validity of , we need to guarantee that is produced from on the basis of the related operations, and the structure of . This will require that each operation is specified properly to meet desired consistency properties (e.g., feasibility), the whole structure (i.e., integration of operations) is consistent, and all of the items (e.g., variables, operators) in formal expressions of the specification are used correctly. Because inspection has to be performed by human inspectors in general, the specification needs to be divided into meaningful parts so that the inspectors can easily grasp the functional behaviors defined, and can concentrate on the inspection of each part of the specification. For this reason, our method takes functional scenarios, and consistency properties (see Section III) as guidelines for carrying out inspections. Specifically, the inspection strategy is as follows. • All functional scenarios defined in the specification must be inspected, but the inspection should, in principle, focus on a single scenario each time. • To inspect a single functional scenario, the inspector should, in principle, first inspect every operation involved in the scenario, and then inspect the integration of the operations involved to ensure that the interactions among them satisfy related consistency properties.

670

A functional scenario (FS) describes a specific behavior in terms of the relation between specific groups of input and output values, and its representation can be different from language to language. In the context of the SOFL language, an FS is represented by a sequence of process sets where a process in SOFL is similar to an operation in VDM: producing a group of output data flows, and/or updating data stores from a group of input data flows and/or data stores. Dynamically, the behavior of the scenario is defined by sequentially executing the process sets involved, where the processes may be executed in in each process set parallel because the order of their executions is not significant. To ensure that the behavior is valid, we must examine each related process to ensure that its specification satisfies related consistency properties, and we must examine the interfaces of the related processes to ensure that their interactions are consistent. Meanwhile, we must also ensure that the statements (including the signature, and the pre- and post-conditions) of each process specification accurately reflect the intended requirements and/or design. Because all functional behaviors are defined by functional scenarios in a specification, ensuring the consistency and validity of every scenario will lead to the assurance of the consistency and validity of the entire specification. Let us consider an automated teller machine (ATM) software system as an example. Figs. 1, and 2 show part of the system specification written in the SOFL specification language. The specification is composed of a Condition Data Flow Diagram (CDFD) in Fig. 1, and the associated module in Fig. 2. The operational behaviors of the system are interpreted as follows. , arWhen a customer’s request, denoted by the data flow rives at the process , this process recognizes the type of the request (for withdrawal or for a balance state. ment), and makes a decision represented by the data flow that The data flow then reaches the process demands the card id, and password from the customer, mod, respectively. (The eled by “read-only” data stores , and “read-only” feature is indicated by the directed line from the store to the process.) The process then checks them against the containing all the customers’ “read-write” data store bank accounts. (The “read-write” feature is indicated by the dito the store.) If the rected line from the process and are confirmed to be correct, the account information is transferred to the process through the data flow , or to the process through the data flow ; otherwise, an error message, represented by the data flow , is provided (all of these three output data flows are exclusively produced). Assume that is made available to the . If is not greater than the amount process available for withdrawal in the account, the requested amount of is delivered; otherwise, an error message is produced and a message is sent to the process . If the customer chooses to retry the service and the total number of his (or her) trial is less than a certain number, the data flow will be generated and sent to the process to repeat the process; otherwise, the process will proto terminate the service. If is duce a message , the data flow made available to the process


Fig. 1. CDFD for a simplified ATM.

Fig. 2. Module associated with the CDFD for the ATM.

is generated. This specification defines four functional scenarios: 1

(freqg; f~accountsg)[fReceive Requestg; fCheck P asswordg;

fW ithdrawg; fCancelg](ferr2; end mesg g; faccountsg) (freq g; f~accountsg)[fReceive Requestg; fCheck P asswordg;

2 fW ithdrawg; fCancelg; fCheck P asswordg;

3 4

fW ithdrawg](fcashg; faccountsg) (freq g; f~accountsg)[fReceive Requestg; fCheck P asswordg] (ferr1g; faccountsg) (freq g; f~accountsg)[fReceive Requestg; fCheck P asswordg;

fShow Balanceg](fbalanceg; faccountsg)

.

All the intermediate data flows (e.g., , , ) are omitted in the scenario expressions for brevity. Each functional scenario clearly describes how the output data flow(s), and the final ) are produced store(s) (e.g., on the basis of the corresponding input data flow(s), and the ini), where detial store(s) (e.g., notes the initial value of the store variable before the , , and execution of the scenario. Note that although are all modeled as data stores that represent data existing


independently of system execution (e.g., files, databases, the user’s inputs), which is indicated by declaring them as decorated external variables in the module (e.g., ext #accounts: set of Account), we only show the data stores that are likely to be , in the scenarios. changed by the system, such as In general, functional scenarios can be derived from a CDFD using a depth-first searching algorithm as indicated in our previous work on test case generation for integration testing based upon SOFL specifications [34]. Beginning with the in starting process of the CDFD, such as Fig. 1, the algorithm will take one group of input data flows (all of them are required for executing the process) and search for one possible group of output data flows of the process. Treating each of the output data flows as a possible input for another process, the algorithm will continue to search for the subsequent processes until one group of output data flows of a of in Fig. 1, is terminating process, such as reached. The result of this search will be a functional scenario, such as Scenario 1 listed above. To find other scenarios, the algorithm will choose another group of output data flows of of , the terminating process (if any), such as which results in another functional scenario, such as Scenario 2 above, until all groups of output data flows of the terminating process are exhausted. The algorithm will then trace back to the preceding process of the terminating process to choose another group of output data flow for a similar searching. This going forth and back searching will continue until all of the data flow groups of all processes in the CDFD are exhausted. When encountering a cyclic data flow, the algorithm will form a cyclic scenario in which the same processes involved are only allowed to repeat once, even if they may actually be executed more than twice semantically. For example, Scenario 2 is a cyclic scenario . Such a treatment resulting from the cyclic data flow will force the inspector to examine the cyclic execution of processes, but avoid dealing with all of the possible executions that may be unaffordable to do in practice. Because a cyclic scenario shares the same characteristic of being a sequence of processes as a non-cyclic scenario, it can be inspected in the same way as for the non-cyclic scenario. SOFL supports structured analysis and design to reduce the complexity of each level CDFD, and to achieve modularity of the specification. This support is fulfilled by decomposing processes into CDFDs, which define in detail how the outputs of the processes are produced on the basis of their inputs via necessary intermediate processes. This feature allows each level CDFD to be designed on a small scale, thus containing a small number of functional scenarios for inspection. Our major concern here is what needs to be done when inspecting a process with a decomposition in a scenario. To let the inspector concentrate on a single task each time for the sake of efficiency and correct judgment, we adopt the following additional inspection strategy. No. 1 Inspect the CDFD for each level, and its associated module specification by focusing on the functional scenarios involved. No. 2 For each process with a decomposition, inspect both the process specification, and its decomposition to confirm their consistency.

671

in scenario 2 above is deAssume that the process composed into a low-level CDFD, and that its associated module . When inspecting the CDFD in is named Fig. 1, and the associated module, the inspector should in principle first inspect all of the four scenarios given above, and then focus on the inspection of the consistency between the process , and its decomposed CDFD. If there are any other processes with decompositions, they will be treated similarly. Two other important issues related to our inspection strategy are the selection, and definition of the consistency properties. We will discuss these issues in Section III, after describing the inspection procedure. B. Inspection Procedure A clear procedure is necessary for a systematic application of the inspection method in practice, and for the construction of a software tool to support it. As Fig. 3 shows, an inspection starts with the derivation of functional scenarios from the specification under inspection. The scenarios together with the related specification information serve as a guideline for deriving inspection tasks to form a checklist. Each inspection task requires a check to determine whether a specific property (e.g., P1) can hold (de), or holds (denoted by ). That noted by a property P1 can hold means that P1 evaluates to true for some values of its free variables, and that P1 holds means that P1 evaluates to true for any values of its free variables. Our major concern is how to help the inspector effectively perform an inspection task. We have developed a graphical notation, known as the Inspection Task Tree (ITT), for visually presenting an inspection task and its decomposition. The principal feature of ITT is that it clearly shows inspection sub-tasks, their relation to the top-level task, and the order in which they need to be checked. It also provides a rule for the evaluation of inspection results. With tool support, as we will see in Section VI, all the specific inspection sub-tasks can be automatically supplied, and the order of their inspection can be managed. Inspection of a specific sub-task can be carried out either through reading the content of the task, if it is simple, or through generating some “inspection cases,” as will be discussed in detail in Section IV-B. After all the sub-tasks have been inspected, the inspection results are evaluated on the basis of the ITT logic to produce an inspection report, which records all the errors identified. III. CONSISTENCY PROPERTIES FOR CHECKLIST As explained earlier, consistency properties are used as a guideline for inspecting specifications in our method. Therefore, it is important to understand what the properties are. We choose those whose violation will affect the validity of the formal specification. Consider the specification composed of the CDFD in Fig. 1, and the module in Fig. 2 as an example. If a is available (i.e., the variable is bound desired data flow to a value in its type by some event), and it satisfies the pre-con, one of the output data dition of the process , , , , and of the CDFD flows is expected to be produced, and to satisfy the post-condition of , , the corresponding process (i.e., , or ). The produced output data flow is

672


them requires a different way of thinking (based upon multiple data flows rather than upon single control flow), which leads to different algorithms for implementation. Another minor reason is that the discussion of these properties will facilitate readers, especially those who are not well familiar with formal specifications, in understanding the rest of the paper. A. Feasibility As explained previously, the feasibility of a process ensures that the output satisfying its post-condition can be produced for any input satisfying its pre-condition. The input here includes values for both the input data flow variables and related external variables, and the output includes values for both the output data flow variables and related external variables. Definition 1: Process is feasible iff it satisfies the condition Fig. 3. Inspection procedure.

also expected to be valid with respect to the designer’s need. To this end, we must ensure the following. • Every process in the CDFD is feasible. A feasible process ensures that for any input meeting its pre-condition, there exists an output that satisfies its post-condition. • The pre-condition of a process must be implied by the postconditions of the preceding processes. Thus, the transformation from the input to the output of the CDFD through the related processes can be realized. Otherwise, there will be a threat to the validity of the CDFD. For example, if the does not post-condition of the process imply the pre-condition of the process , it that satwill suggest the existence of some value for , but violates isfies the post-condition of . Thus, the pre-condition of will guarantee neither the existence of a value for the data , nor a valid . flow • A process with a decomposition (another CDFD) must ensure that a desired input meeting its pre-condition is consistently transformed into a desired output satisfying its post-condition via its decomposition, because its decomposition is expected to refine the function of the process. On the basis of these considerations, we define four categories of consistency properties: feasibility, invariant-conformance consistency, integration consistency, and decompositional consistency. The fundamental ideas of these properties may not all be novel compared to those used in existing work, such as VDM [24], PVS [5], B-Method [32], Cogito [35], [36], specification animation [37], and the work by Jaffe et al. [38], but we need to formally describe them in this section for two reasons. One reason is that these properties are formed based upon a data flow model rather than upon a control flow model as in the existing work. In the case of a property concerned with only an individual operation (or process in SOFL), such as feasibility or invariant-conformance consistency, its definition has no substantial difference from the existing work. However, for the properties concerned with the structure of the specification, such as integration consistency and decompositional consistency, their expressions differ from those in the existing work, and forming

where denotes the predicate resulting from substituting (the initial value of external variable before the execution of ) for all the free occurrences of the external variables of (the final value after the execution . Because the same external variable in of ) in the pre-, and post-conditions denotes the initial, and final values before, and after the execution of , respectively, we must use instead of in the above condition to keep the use of the external variable consistent. To help you understand this concept and other related properties to be defined later, we provide a simple but adequate example. We could pick up a process of the CDFD in Fig. 1 as an example, but this is difficult here, because its pre- and , post-conditions are either too simple (e.g., ) to be adequate for the illustration, or too , ), due to the complicated (e.g., , to convey involvement of the complex data store the main idea succinctly. We therefore consider a process describing a fundamental operation in software systems as an example, which is specified as

The process takes a value of type (pre-defined), . If appears in , and searches for it in the sequence will remain unchanged; otherwise, is appended to the end of . In either case, the index of in is given as the output . Then, the condition for the feasibility of of is


B. Invariant-Conformance Consistency The invariant-conformance consistency requires that each invariant not be violated by the pre- and post-conditions of related processes. That is, when the pre- or post-condition of a related process evaluates to true, the invariant must hold. There are two kinds of invariants: type invariant, and external variable invariant. A type invariant imposes a constraint on the values of the type, and requires it to be sustained by every process in the system. An external variable invariant is a constraint on the values which the external variable can take, and must be sustained by every process. We define and discuss the invariantconformance consistency for the two cases in turn. Let us start with the type-invariant-conformance consistency. be Definition 2: Let an invariant on type , where represents the free variable(s) occurring in . Then, process , and are consistent iff the following conditions hold. , and 1) 2) , where

and the expression junction

673

that, because a decorated variable like in the post-condition of process denotes the initial value of the external variable before the execution of , there is no need to check whether it violates the invariant after the execution. That is why we above. exclude all the decorated variables in the variable set Condition 1 in Definition 2 states that when the pre-condition ) must hold for of is satisfied, the invariant body (i.e., any variable of type involved in the pre-condition, and condition 2 describes a similar requirement for the post-condition. be defined as negative For example, let the type integers by the type declaration

and the type invariant

Applying Definition 2 to the process above and the invariant, we obtain the following conditions for type-invariant-conformance consistency. 1) 2)

represents the con-

assuming that , and each is a variable of type . This formula states that variables of type in the pre-condition or post-condition must , satisfy the type invariant . The functions , and are defined as the equation at the , bottom of the page. In the definition of to denote the set of all free variables we use to represent the set occurring in predicate , and of all the element variables of the compound type variables appearing in . By a compound type variable, we mean a variable representing a value of set, sequence, map, composite, product, or union type in the context of the SOFL specification language. Because creating a formal expression of is tedious on account of the different syntactic structures for different compound type variables, we omit it for brevity. Note

Next, we define the variable-invariant-conformance consistency. Definition 3: Let an invariant on external varibe , where ables is a predicate expression. Process , and are consistent iff the following conditions hold. 1) 2) Condition 1 in Definition 3 states that when the pre-condition must hold, enevaluates to true, the invariant suring that the truth of the pre-condition will not cause the violation of the invariant; otherwise, it will imply that either the invariant contains errors, or the pre-condition is inappropriate. Condition 2 in Definition 3 means that when the conjunction of the pre-condition, the invariant before the process (denoted by ), and the post-condition all hold, then the inafter the process must also hold. variant

if predicate otherwise if is a decorated variable otherwise if the type of variable is otherwise

does not contain variables of compound types

674


D. Decompositional Consistency

Fig. 4. A simple CDFD.

Consider the process as an example. Assume that the following additional invariant is imposed on a single : external variable

Then, the application of Definition 3 to the process and the invariant results in the following conditions for variable-invariantconformance consistency. 1) , 2)

C. Integration Consistency Integration consistency is intended to ensure that the pre-condition of a process in a CDFD is always satisfied by its input data flows and the related data stores. The pre-condition of the process defines the responsibility for the operational environment that supplies the input data flows and stores. To ensure this property, the post-conditions of the preceding processes of the process must imply its pre-condition, because the preceding processes produce the input data flows, and may update the related data stores for the process. Definition 4: Let

be processes that produce the data flows , respectively. Suppose stitute sufficient input data flows to enable process satisfies the condition below, we say that process the integration consistency property.

con; then if satisfies

Let us take the processes , , and in Fig. 4 as an example. For , , and to satisfy the integration consistency property, three conditions must be satisfied: 1) 2) 3)

Decompositional consistency is a property concerned with a process, and its decomposition. A process is consistent with its decomposition iff for any input satisfying the pre-condition of the process, the output produced by the decomposition satisfies the post-condition of the process. The process is treated as a functional abstraction of the decomposition. Formally, the consistency property is defined as follows. Definition 5: Let process be decomposed into the CDFD . is consistent with iff they satisfy the condition

In this definition, is treated as a relation between input data flows together with initial values of the external variables and output data flows together with the final values of the external variables

Here, , and denote the types of variables , and , respectively; and , and denote the types of variables , and , respectively. To inspect whether this condition is satisfied, we need to check whether the following condition holds: for every possible input-output scenario of process , there exists a functional scenario in that takes the same input meeting the pre-condition and generates the same output satisfying the post-condition of . Formally, let

meaning that

contains

scenarios, and each scenario states that output is generated from input through . Let

meaning that

has

scenarios, and each scenario

states that input is transformed to output through a sequence of process sets . We use denote the conjunction of the pre-conditions of all the processes in process represent the conjunction of the post-conset , and ditions of all the processes in . The task of inspection for the decompositional consistency is to check the following three conditions. 1) Each scenario is inspected, where , 2)

for all


675

has two functional scenarios, expressed in terms of input-output relations: , 1) 2) . Two functional scenarios are derived from the decomposition of process : , 1) 2) . To inspect whether the decomposition is consistent with process , we need to 1) inspect the scenario

1.1)

check

the

condition

Fig. 5. Example of process decomposition.

1.2) check the condition 3) for

all

,

Condition 1 requires that all the processes in each functional scenario and their interactions be inspected. Condition 2 requires that the corresponding sub-expressions of the pre-condition of be weakened by those of the starting-processes . A , sub-expression of a pre- or post-condition, such as that is concerned with means a constituent predicate of of variables , and . Assume that the pre-condition in Fig. 5 is defined as process . Then, , and . Condition 3 states that each sub-expression (defining an output data flow and the final external variable) of the post-condition of must be implied by the conjunction of the pre-post conjunctions of all the processes in the corresponding scenario. The pre-post conjunction of a process here refers to the conjunction of its decorated pre-condition and post-condition. Because each process of the functional scenario may change the value of external variable (without changing its name), and the scenario may contain more than one process set, only using tilde to decorate the external variable (i.e., ) is insufficient to keep the appearances of in Condition 3 consistent. To address this problem, we adopt suand for , thus allowing us to use , and perscripts to denote the initial value, and the final value of before, and after any process in , respectively. In this notation, , and occurring in the pre-, and post-conditions of any process in are assumed to appear as , and , respectively, and apparently that (occurring in the post-condition of the last process of the functional scenario) will be the same as used set of Condition 3. Note that the introduction of in this notation in Condition 3 is merely for the purpose of consistently expressing the condition; it does not affect the way of writing the pre- and post-conditions of any process using decorated variables like in the original formal specification. For example, suppose that process is decomposed into , , , and , as the CDFD containing processes takes either , or , but not both; shown in Fig. 5. Process and generates both , and , as well as writing to store . It

Because the pre-condition of and is likely to inor , which is constrained volve data flow variable by the post-condition of , the pre-condition of is likely to involve data flow variable that is constrained by , and the post-condition of is the post-condition of likely to contain input variable . The pre- and post-conditions of all the four processes are generally necessary in that is likely to contain implying the post-condition of and . both input , and outputs 2) inspect the scenario

2.1)

check

the

condition

2.2) check the condition

Note that because process has no relation with store , is equivalent to (not involving ), and is equivalent to . The definitions of the consistency properties above have established the formal basis for our inspection method, so we now turn our attention to managing the inspection process via inspection task trees. IV. INSPECTION TASK TREE (ITT) ITT results from a development of the Fault Tree Notation for safety analysis [39], achieved by attaching the “gates” to “event nodes,” and by extending the “gates” with new types. Such a development makes ITT suitable for the presentation of inspection tasks in a systematic, logical, and hierarchical manner. Fig. 6 shows the major components of ITT. Each node of an ITT represents an inspection task, defining what to do about a property; it may be connected to “child nodes” in different ways,

676


Fig. 7. Rules for constructing ITTs from inspection tasks.

Fig. 6. Major components of ITT.

A. Construction of ITTs for Inspection Tasks

depending on the type of the node. There are two kinds of inspection tasks: one to check whether the property holds, and the other to check whether the property can hold. The former is represented by a rectangular node, while the latter is represented by a round-edged rectangular node in ITT. , iff all Definition 6: A property holds, denoted by the components of are valid, and evaluates to true for any values bound to all of its free variables. can hold, denoted by Definition 7: A property , iff all the components of are valid, and evaluates to true for some values bound to all of its free variables. , and resemble the terms The terms “tautology” and “satisfiable,” respectively, in predicate logic, requires not only that but they are not the same. be a tautology, but also that its components (e.g., variables, operators, and constants) should all be valid (with respect to the designer’s desire that are supposed to be understood by the inrequires not only that be satspector). Similarly, isfiable, but also that its components should all be valid. For exof the ample, to judge whether the pre-condition can hold (i.e., process is true), we need to check whether there exists a sequence of its type such that the predicate evaluates to true; we also need to check whether the operator is valid is valid (e.g., should it be another operator?), the variable (e.g., should it be a different variable?), the operator is valid (e.g., should it be ?), and the constant 0 is valid (e.g., should it be 1?). Note that an ITT is not only used as a graphical representation of a textual predicate expression, but more importantly as a tool to guide the inspection process by showing what needs to be inspected in turn.

An ITT for a given inspection task is constructed by applying relevant rules, each of which shows how a large task is decomposed into sub-tasks. Fig. 7 shows a list of rules for constructing ITTs and their semantics. , the For example, to ensure the task corresponding ITT shows that we must first inspect whether and can hold, respectively, where represents the free variable(s) occurring in both predicates and , and then inspect whether there exists some value in the type of such that the conjunction can hold. Of course, without it is not wrong to merely inspect and , independently, first inspecting but our experience shows that first inspecting and , independently, and then inspecting their conjunction as a whole has three potential benefits. Firstly, it gives the inspector an opportunity to analyse whether the components and are valid (on a smaller scale each time). Secondly, the truth of can be judged by simply concentrating on the free variables shared by and , ignoring . Finally, such an inspection strategy other aspects of has the potential to improve inspection efficiency: at the point or is of understanding that either or is not satisfiable or not valid), false (implying that the inspector can draw the conclusion that the top-level task is false. The rules for inspecting quantified expressions are also interesting. These rules present a technique that allows us to make good use of the established rules for non-quantified expressions, and consequently avoid inspecting quantified expressions directly (which are rather difficult to manage in many cases). These rules are constructed on the basis of the following logical relations. 1) 2) 3)


677

TABLE I EXAMPLE OF INSPECTION CASES

In our previous research on specification testing [41], we established a set of criteria for the generation of test cases. Those criteria can be utilized to generate inspection cases. For example, to inspect whether

Fig. 8. ITT for inspecting the type-invariant-conformance property.

We give an example to illustrate the process of building an ITT. Consider the type-invariant-conformance consistency of the given in Section III. It is converted process into the following disjunction by removing the implication, and the universal quantifier

The inspection task is then

is true, we generate some inspection cases to make every disjunctive clause true and false once each, as shown in Table I, and the validity of the property is finally judged on the basis of the inspector’s analysis. The important thing in such an inspection is not only the activity of evaluating the expression with the inspection cases, but also the process of generating the inspection cases, and analyzing the expression with them. For example, through an inspection of the ITT in Fig. 8, we found that the is actually not valid with respect to pre-condition . The the functional requirement for the process valid pre-condition should be , because the input of type can also be appended to even when is an empty sequence. C. Inspection Evaluation

The ITT for inspecting this task is shown in Fig. 8. It is generated as by repeatedly applying the translation rule for shown in Fig. 7. B. Reading Techniques An ITT shows how to inspect a top-level task by carrying out all of the atomic inspection tasks. Atomic tasks inspection requires certain reading techniques, which we discuss in this section. For each atomic task inspection, we adopt the following thee reading techniques. • Check all the related constants, variables, types, operators, and their combinations to analyse whether they are used properly. • For a simple task, perform an ad hoc reading of its content. • For a complex task, inspect it with certain values for the free variables, which are called inspection cases in this paper. The characteristic of inspection is that errors are ultimately identified by the inspector, and the right decisions on errors require the inspector to understand the content of the task [40]. ) can Because the content of a simple task (e.g., be easily understood by the inspector, potential errors can be detected easily. However, for a task with complex content (e.g., ), a reading can be adopted. technique called Using this technique, we first generate inspection cases, and then analyse the predicate with them.

The goal of an inspection based upon an ITT is to decide whether the check of the top-level task reveals errors, and provide an inspection report containing descriptions of the errors detected (if any). Such a decision can be reached by proper evaluation of the inspection results of all the atomic tasks based upon the concept of a minimal cutset [39]. A minimal cutset of an ITT is the minimum number of atomic tasks whose truths lead to the truth of the top-level task in the context of predicate logic, which is adopted in the semantics of ITT. Let us take the ITT in Fig. 8 as an example. The relation between the top-level task and the atomic tasks is expressed in the form

which means that the truth of the top-level task is determined by the truth of one of the five minimal cutsets connected by the operator “ ,” where “ ” means logical “or”. Note that the “ordered or” gate in the ITT is treated as the logical “or” in evaluating the truth of the top-level task, because the “order” is only intended to guide the inspector in undertaking the inspection tasks. The result of inspecting a top-level task is one of the two

678


judgments (indicating no error is found), and (indicating some error is found), together with a description of the errors detected. V. CASE STUDY We have conducted a case study of our RIM by applying it to the inspection of an Automated Teller Machine (ATM) software specification. The purpose of this study is to allow us to learn about its performance, on a relatively small scale, in terms of usability and effectiveness in finding errors, to find potential challenges in using it, and to investigate how RIM can be well supported by a software tool. The description of the case study in this section is intended to share our experience with readers so that they may benefit from it somehow in their own research or application. A systematic and rigorous evaluation of RIM would require a large-scale formal specification, a large number of well-trained subjects, and a well-planned and monitored inspection process. Such a study is beyond the scope of this paper.

The target specification contains 223 injected errors. The errors can be divided into eight categories based upon their nature: variable (55), type (38), internal consistency (48), invariant-conformance (10), satisfiability (8), integration consistency (19), validity (29), and others (16). The errors in the category of “variable” include wrong variable declarations and inconsistent uses. The errors concerned with “type” include wrong type declarations, wrong type names, and inconsistency with the associated type invariants. The errors in the category of “internal consistency” include inappropriate uses of variables in the pre- and post-conditions that violate the relevant semantic constraints of the SOFL language. The errors in the categories of “invariant-conformance” and “feasibility” of processes, as well as “integration consistency,” are those that violated the corresponding conditions defined in Section III. The errors in the category of “validity” include the wrong use of operators (e.g., “ ” should be “ ”), wrong use of types in declaration of data flows or stores, and logical mistakes in process specifications ” should be “ ”). (e.g., “ The remaining errors are classified into the category “others”.

A. Background We invited three senior students in computer science to join the study as subjects. The three subjects were asked to inspect the same specification of an ATM within the same time, where the time planned for the inspection was decided on the basis of our previous experience in inspecting an IC card software system of similar scale. The ATM specification was written in SOFL, and was unknown to the three subjects before the case study. The specification defines the following six functions: authentication, operations on current accounts, operations on savings accounts, money transfer between accounts, operations on foreign currency accounts, and changing a password. The specification is 69 pages long, and its details are described in a technical report [22]. The ATM system was selected for the case study mainly on the basis of its critical nature, commercial impact, scale, and the availability of its formal specification in SOFL. All of the three subjects used RIM to carry out their inspections independently. Because this case study was intended to allow us to observe the performance of RIM in the situation where no tool support is available, the three subjects did not use our tool described in Section VI for the inspection, but used some text editors on their laptop to manage their documents. According to MacDonald and Miller’s study [42], there is almost no difference between the results of tool-based and paper-based inspections in terms of error detection effectiveness. Therefore, the way of applying our RIM in this case study did not affect our observation of its performance. To help us measure the inspection effectiveness, we also invited an independent person (a research student) to inject errors before the inspection started. The person was not aware of our inspection method at that time, and was not told what the specification with injected errors, which is called the target specification, would be used for. Furthermore, the error injection was carried out on the basis of the considerations of error distributions, specification situations (e.g., what kind of errors can be injected into a specific part of the specification), and his previous experience of injecting errors for other experiments.

B. Inspection Process and Results The inspection process included the following activities. We first made a preparation for the inspection, including training the subjects in our RIM, injecting errors, and designing inspection schedule and activities. We then held a meeting corresponding to the “overview” in Fagan’s inspection process [16]. At the meeting, the author of the original specification explained the major functions and the structure of the specification to all of the subjects. After that, the three subjects worked independently to inspect the target specification. The inspection conducted by each subject was divided into sessions, each of which was limited to ninety minutes to fit into a class slot, and conform to Fagan’s suggestion (no more than two hours for an inspection session) [16]. After the completion of the inspections by all of the subjects, the inspection results were collected and analysed by the person who injected the errors and the author of the specification. We then calculated the error detection effectiveness, using the formula Detection effectiveness number of (injected) errors detected total number of errors injected The inspection results of the three subjects were obtained by specific inspection tasks, which were formed on the using basis of 188 consistency properties derived from the specification, as shown in Table II. Each row in the table shows the results of all the subjects’ inspection, including the average result (in the column of av), and the individuals’ results (in the columns of 1, 2, and 3.) The total average detection effectiveness of the three subjects is 85%, which is reasonably good compared to other experiences, and experiments reported in the literature. C. Experience and Discussion Our observation and analysis suggest that the major strength of our method consists in the rigorous, systematic process for deriving, and then inspecting consistency properties, because


TABLE II INSPECTION RESULT

679

SOFL module specification, but VDM uses constructs in programming languages (e.g., sequence, selection, and iteration) for the integration of operations while SOFL uses condition data flow diagrams for operation (or process in SOFL) integration. Our method can also be applied to inspect UML designs in which UML activity diagrams are used to graphically describe a system’s behaviors, and OCL is used to define the functions of objects involved in the activity diagrams using pre, and post-conditions. With necessary modification, the rigorous inspection method can be applied to inspect designs in UML OCL because of their similarity to SOFL activity diagram specifications. VI. PROTOTYPE TOOL

the subjects were effectively guided so that they could concentrate on the analysis of specific inspection targets. The ITT notation is intuitive for presenting the inspection tasks, and effective for evaluating the inspection results. We also found two challenges in using the method. One was that the derivation of consistency properties and inspection tasks by the subjects was very difficult, without the help from the author of the specification. The reason is that although the three subjects were familiar with the SOFL specification language, they lacked knowledge about the consistency properties, and how they are formed. In our case, the author had to work together with them in forming the inspection tasks. With appropriate tool support, this problem can be easily addressed. Another challenge is that the subjects sometimes found it very difficult to determine errors. The major reason for this was that the subjects might not understand the meaning of the related expressions of the specification, either because they are too complex, or because the subjects lack sufficient ability to concentrate on digesting them. Even if they understood the meaning, they might sometimes not understand whether the formal expressions accurately reflect the true needs. This problem, however, remains in all static verification and validation techniques, though to varying degrees. A potentially effective way of addressing this challenge is to let the author of the specification, the client, and the inspector work together in such a manner that the inspector raises questions, and the author tries to answer them. During the “question time” process, errors can be detected. Although the “question time” technique can be applied to any kind of conventional inspection techniques, our method offers a rigorous, systematic process for ensuring that no part of the inspection target is missed, so that the technique can play a more effective role in finding errors. It is important to understand that the underlying principle of our inspection method is not limited only to the SOFL specification language, but can be applied to any model-based specification languages, although the definition of the consistent properties may vary from language to language. For example, applying our method to inspect VDM specifications can use almost all the consistent properties defined for SOFL specifications, except the decompositional consistency property. The reason is that the style of VDM module specification is inherited by

We have built a prototype tool, called the Rigorous Inspection Method Tool (RIMT), to support RIM. The goal of building the tool is to investigate the feasibility of automating the method, which is expected to establish a foundation for developing a mature product in the future. In this section, we first introduce the major functions of the tool, and then evaluate its features by comparing it to related tools. A. Major Functions The tool is intended to support the five activities for inspection that are required by our method, as shown in Fig. 3 in Section II-B. Specifically, it provides the following potential functions. • graphical editor. This allows the user to freely draw and edit ITTs, for example, by deleting, moving, enlarging, and naming tasks. It also has the ability to prevent the user from drawing or creating unclear structures. When a node (representing an inspection task) is moved from one place to another, all of the related parts of the ITT are ensured to move coherently while preserving the structure of the ITT. Fig. 9 shows a snapshot of the graphical user interface (GUI) of the tool. • Automatic syntax, type, and internal consistency checking. The internal consistency of a process ensures that the use of variables in its pre- and post-conditions does not violate the relevant semantic constraints of the language. For example, an output data flow variable cannot appear in the pre-condition, and all of the free variables used in the preand post-conditions must be declared in the process signature. Because all of these three aspects of a specification are decidable on the basis of the syntax of the SOFL specification language, the tool can check them automatically. • Automatic derivation of inspection tasks for the checklist. Given a SOFL specification composed of a CDFD and a module, the tool can derive inspection tasks for the checklist from it. This result is realized by first deriving all of the functional scenarios from the CDFD, and then deriving all related consistency properties along each scenario. The module is needed for deriving the consistency properties because it contains all of the type invariants and process specifications that are necessary for expressing the properties. Based upon the properties, all of the inspection tasks are finally formed, and organized according to the functional scenarios. When a scenario is selected, the related

680

•

•

•

•


inspection tasks will be displayed in the GUI in a form suitable for selection. Once the inspection of a task is completed, the tool will automatically mark it to indicate its current status. Automatic construction of an ITT for an inspection task. When a specific inspection task is selected from the checklist, the tool will automatically construct an ITT on the basis of the translation rules defined in Fig. 7. For example, the ITT in Fig. 9 is generated to inspect the type-invariantdeconformance property of the process fined in Section IV-A. Tool-guided inspection support. The tool guides the inspection of the ITT by dynamically highlighting the current inspection task (a task box) as the inspection proceeds. The inspector then analyses the task, and submits his or her inspection result through a small window, called the submission window, as shown in the top right corner of Fig. 9. If the result includes any detailed description of errors detected, the tool will display it in a well-organized format as part of the inspection report, which is also stored in a file for further analysis or error correction. When the inspection of the current task is completed, the tool will automatically highlight another task for inspection based upon the ITT logic. This process continues until all the necessary tasks are inspected. Automatic inspection result evaluation. On completing the inspection of all the necessary atomic tasks in the ITT, the tool will automatically evaluate the inspection results on the basis of the ITT logic to produce an inspection report. The report includes a final conclusion, “false” or “true,” indicating whether the inspection has revealed errors, or not, respectively. When the conclusion is “false,” the inspection result usually includes a list of error descriptions recorded during the inspection process. Automatic hyperlinks among properties, ITTs, and specification components. To help the inspector find and understand the related contents of the current inspection task, the tool is expected to provide hyperlinks to allow him or her to quickly link a component of an ITT (e.g., an expression in an inspection task) to its corresponding property contained in the checklist, to link a property in the checklist to the related components (e.g., processes, invariants) in the specification, and to link a component of the ITT to the relevant expressions in the specification.

B. Evaluation Our RIMT exhibits several interesting features in supporting inspection of formal specifications, some of which may be shared by existing tools for similar purposes (but not the same), such as VDM Toolbox [43], [44], and PVS [5], [45]. To explore and evaluate the distinct features of RIMT, we have applied RIMT, VDM Toolbox, and PVS system to “verify” a module, , of the ATM softcalled ware specification in SOFL that was developed in our previous project [22]. The module contains the declarations of four types and three store variables, one invariant, and eight processes. The module is associated with a CDFD describing how the processes are integrated based upon data flows and stores. Through this case study, we have found several important points.

Fig. 9. GUI snapshot of the support tool.

Firstly, the objectives of the tools are different. Although all of the three tools can automatically derive the consistency properties to support verification of formal specifications, they support different philosophies. PVS supports formal proof of theorems. A consistency property can be described as a theorem, and its validity can be justified through formal inferences. VDM Toolbox supports type checking of specifications. It also supports specification interpretation by evaluating its pre- and postconditions based upon selected input values. Our RIMT supports rigorous inspection, a static analysis technique, of formal specifications in a systematic manner. The technique emphasizes the cooperation of machine and humans in that all of the preparation activities of an inspection are performed by the computer, while the inspection process and judgments of errors must be made by humans with a clear guidance from the tool, thus allowing the examination of both the consistency and the validity of specification expressions. Secondly, the usability of the tools are different. All of the three tools have a GUI, but the one of PVS tool is poorer than those of VDM Toolbox and our RIMT. Furthermore, using PVS for formal proof requires the user to have high skills in logic, proof theory, and techniques for carrying out proofs, because a proof process is an intellectual activity that cannot usually be performed automatically. Proof sequences produced by the tool are not easy for humans to understand, and in particular when a proof fails, finding the problem based upon the generated proof sequence is very difficult. In contrast, VDM Toolbox is easier to use, but it requires mathematical functions to be defined explicitly (for executability). Furthermore, test cases for specification interpretation must be generated by the user, without any clear guidance from the tool. The tool also does not support static inspection, although the automatically generated properties may be used for inspection. Our tool is the easiest one to use. Given a SOFL specification, the tool will automatically generate a list of consistency properties. When a property is selected, an ITT will be automatically constructed. The inspection of the tasks in the ITT will be carried out by the user under a clear, comprehensible guidance from the tool (based upon the ITT logic). The inspection process does not require the user (inspector) to apply the knowledge of proof, but to focus on the validity of atomic expressions involved in properties. Thirdly, the capabilities of the tools in supporting the visualization of verification are different. The visualization of verifi-


cation may not be important in theory, but extremely helpful in engineering due to its superior intuitivity. The VDM Tools do not support any visualization of the verification process. The PVS tool helps to build a visualized proof tree for a formal proof, but generally the tree must be constructed by the user, with the help from the tool, because its content is usually unlikely to be decided automatically by the tool. Our RIMT supports the visualized ITT notation, and inspections based upon it. For each selected inspection task, the tool automatically constructs an ITT to divide the task into smaller sub-tasks to support the “divide and conquer” strategy for inspection. Although the ITT approach shares the same principle of the fault tree analysis [39] that has been applied to safety analysis of software artifacts (e.g., program, design) by Leveson [46], [47], it differs from Leveson’s work in two aspects. First, the philosophy of building an ITT for an inspection task differs from that of building a fault tree from a program. An inspection task is an expression of whether a consistency property can hold or holds, and the property itself is a predicate expression. The rules of constructing an ITT, which is one of the major contributions in this paper, are established on the basis of predicate logic, while the rules of building a fault tree are based upon the operational semantics of program constructs (sequence, selection, and iteration). Inspections based upon ITTs focus on the validity of related expressions, while fault tree analysis of program constructs concentrates on the safety concern (i.e., whether the program can cause safety problem of the system) in Leveson’s work [46]. Second, the ITT notation has improved the syntax of the fault tree notation by attaching logical connectors (or “gates” in fault trees) to “task” boxes (or “event” nodes in fault trees), thus saving space that would be created by drawing “gates” and their connected lines. Finally, the maturity of the tools are different. The VDM Tools and PVS tool have both been developed into a mature tool through a long period of effort in testing and evolution, while our tool is currently a prototype system built for research purpose. The prototype tool is successful in the sense that it has helped us explore the feasibility of automating all the important activities suggested by our inspection method. To develop the prototype into a practical software product, we will have to carry out more testing, and extend it to support other activities as well. Those activities include discussions between an inspector and the author of the specification, management of the inspection process, error elimination, and follow-up checking after error corrections. VII. RELATED WORK In this section, we describe related techniques and compare them to our RIM. Parnas and Weiss’s Active Design Review (ADR) proposes an innovative approach to inspecting design documents [48]. The principal innovation of ADR is the use of questionnaires to give the inspectors better defined responsibilities, and to make them play a more active role by getting them actively involved in the design procedure. The overall inspection task is divided into smaller, more focused tasks, and each inspector is required to focus on a specific inspection task within his or her expertise to check for errors of a specific kind, such as inconsistencies or ambiguities, or to check whether the design document has a specific kind of property, such as being

681

well structured, simple, or flexible. Knight and Myers [17] suggested an improved inspection technique called “phased inspections” that consists of a series of phases. Each phase is intended to ensure that the product under inspection possesses either a single property, or a small set of closely related properties. The inspectors in a given phase are responsible for ensuring that the properties defined for that phase are fully checked. While the two techniques above constitute a “divide and conquer” strategy for inspection, our inspection method applies the strategy in creating and presenting inspection tasks on the basis of precise definitions of consistency properties, and in rigorously guiding the inspectors to fulfill each task, and systematically evaluating inspection results. Peer review is a commonly used quality assurance activity in industry [49], and reflects an administrative strategy for review, according to which that a software artifact is reviewed by a person other than the creator who understands the artifact to some extent. Our inspection method focuses on the technical procedure of checking a specification. It can be applied either by the author of the specification, or by peers, and individually, or in a group. Reading techniques play an important role in facilitating defect detection in software artifacts. Ad-hoc reading [50], and checklist-based reading [16], [40], [51], are reading techniques frequently used in industry. During an ad-hoc reading, a software product is simply given to inspectors without any direction or guidelines on how to inspect it, or what to look for. Checklists offer stronger support than ad-hoc reading in the form of questions concerning various quality issues which inspectors are required to answer while reading the document. Scenario-based reading [52], [53], [54] provides guidelines for inspectors based upon scenarios for performing inspections, where a scenario may be a set of questions or a more detailed description of how to carry out inspections. Different scenario-based reading techniques were proposed according to several approaches for developing scenarios: defect-based reading based upon possible defects in the work product [54], [55], usage-based reading, which helps inspectors to focus on the most important parts of a software artifact from a user’s point of view [56], perspective-based reading for inspecting work products from the viewpoint of various roles (e.g., analyst, designer, tester) in software development [53], [57], and slicing-based reading for inspecting individual slices of code [58]. Reading by stepwise abstraction, a reading technique advocated by the cleanroom community [59], [60], requires an inspector to read a sequence of statements in the code, and to abstract the functions which these statements compute. Our inspection method proposes a property-based reading technique that uses the same checklist-based reading and scenario-based reading as the above reading techniques, but differs from them in the contents of inspection questions, and in the formalization of the inspection process for tool supportability. Some researchers have explored the possibility of automating validity or consistency analysis of designs or specifications. Heyer proposed a tool-supported approach to inspecting the validity of software design represented in a subset of UML [20]. The approach is to define assertions (pre- and post-conditions) for use cases representing requirements and sequence diagrams

682

representing design; and then to automatically derive questions using a tool for inspection. The goal of the inspection is to check whether a sequence diagram satisfies the corresponding use case diagram by analyzing whether they satisfy the refinement obligation (weakening pre-condition, and strengthening post-condition). Heitmeyer, Jeffords, and Labaw developed an automated technique for consistency checking of specifications in the Software Cost Reduction (SCR) tabular notation [61]. A verifier for analyzing some important properties, such as timing and security, is provided, and the technique and the toolset mainly supports automatic checking of some syntactically decidable consistency properties, including proper syntax, type correctness, completeness, initial values, reachability, disjointness, coverage, and non-circularity. Leveson, Reese, and Heimdahl described a system engineering environment called SpecTRM in [62], which supports the specification and development of safe systems and software. More specifically, the environment supports the language called SpecTRM-Requirements Language (SpecTRM-RL) to construct system and requirements specifications, and to support quality assurance through formal validation tools, inspections, and simulation. Our inspection method shares the same principle as the work introduced above in supporting the automation of the inspection process, but differs from them in specific techniques. In comparison with Heyer’s work, our method focuses not only on refinement obligations (corresponding to the decompositional consistency in our method) for validity of designs, but also on other properties such as consistency. Due to the precise definitions of the consistency properties, our method allows automation in the formation of inspection tasks, guidance in the inspection process, and evaluation of inspection results. In comparison with Heitmeyer’s and Leveson’s work, our method offers a complementary technique, focusing on rigorous, systematic, and intuitive semantic analysis which may not be automatically performed. Formal proof supported by theorem provers, such as the B-Toolkit for B [32], Z/EVES for Z [63], and PVS [5], is regarded as the most rigorous approach for verification. While its application to small-scale safety-critical systems may be justifiable, formal proof of large-scale, complex systems is challenging. Changes in requirements and documentation during a software engineering process are an inherent feature, and revising formal proofs would be both difficult and costly. Using theorem provers usually requires a high level of expertise (e.g., in selecting proof tactics and inference rules); and proof processes are usually time-consuming, and perhaps not necessarily as effective as existing techniques in design and implementation phases, as stated by Hall: “Proof is no more a guarantee of correctness than testing, and in many cases far less of one” [10]. Hence the approach is little used in industry [64]. Although many opinions about the utility of hand proofs exist, it is widely believed that hand proofs are difficult to formalize using a mechanical theorem prover because the concerns of human-readable proofs are rather different from the precise formulation of mechanized theories. Our inspection approach is aimed at providing a rigorous but easy-to-use technique for ordinary software engineers to improve the conventional inspection process in software engineering. It differs from the


commonly understood hand proof approach in a number of ways. Firstly, it does not require the selection of proof tactics and inference rules, but requires checking of specification properties through checking their constituents. Secondly, the focus of an inspection is not only on logical justification, but also on the validity of the constituents of properties. Finally, inspection is not aimed at establishing the correctness of specifications, but at detecting errors. VIII. CONCLUSIONS AND FUTURE RESEARCH We have described a property-based rigorous method for inspecting model-based formal specifications called the rigorous inspection method (RIM). RIM results from the combination of formal analysis and conventional inspection. Formal analysis is utilized to offer precise guidelines for deriving inspection tasks from specifications, to provide rules for presenting inspection tasks, and to serve as a foundation for evaluating inspection results. Inspection based upon a checklist is used to check every operation (or process) on each functional scenario, and the interactions between those operations. To achieve the qualities of preciseness, visualization, and simplicity of RIM, we developed a graphical notation called Inspection Task Tree (ITT) for presenting the decomposition of an inspection task into atomic sub-tasks, and for guiding the inspection process. We have conducted a case study to obtain an initial understanding of the effectiveness and usability of RIM, and to learn about potential challenges when it is applied in practice. We have also built a prototype tool called RIMT to demonstrate the supportability of RIM. The tool helps to enhance inspection efficiency by automating the derivation of inspection tasks, constructing ITTs, guiding the inspector in carrying out inspections, and evaluating inspection results. In the future, we plan to do more studies on our RIM by conducting systematic comparisons with other existing work in practical software engineering projects. We will continue to develop RIM, and its support tool RIMT to offer more functions, to enhance the level of automation, and to improve its capability of providing intelligent guidance for inspections. We are also interested in developing a systematic technique for effectively integrating inspection, formal proof, and testing approaches; and in studying its effects on the assurance of software quality in practice. ACKNOWLEDGMENT We would like to thank all of the students involved for their contributions to the case study and to the prototype tool. We would also like to thank A. Galloway for reading the manuscript and suggesting ways to address several important issues. Finally, our gratitude goes to the three anonymous reviewers for both their critical and constructive comments, which have helped us improve the quality of this paper. REFERENCES [1] D. Craigen, S. Gerhart, and T. Ralston, Industrial Applications of Formal Methods to Model, Design and Analyze Computer Systems: an International Survey. , USA: Noyes Data Corporation, 1995. [2] G. Babin and F. Lustman, “Application of formal methods to scenariobased requirements engineering,” International Journal of Computers and Applications, vol. 23, no. 3, pp. 141–151, 2001.


[3] S. Sahara, “An experience of applying formal method on a large business application,” in Proceedings of 2004 Symposium of Science and Technology on System Verification, National Institute of Advanced Industrial Science and Technology (AIST) (in Japanese), Osaka, Japan, Feb. 4–6, 2004, pp. 93–100. [4] B. W. Boehm, Software Engineering Economics. : Prentice Hall, 1981. [5] S. Owre, J. Rushby, N. Shankar, and F. V. Henke, “Formal verification for fault-tolerant architetures: prolegomena to the design of PVS,” IEEE Trans. Software Engineering, vol. 21, no. 2, pp. 107–125, 1995, IEEE CS Press. [6] E. M. Clarke, O. Grumber, and D. Peled, Model Checking. : MIT Press, 2000. [7] T. Miller and P. A. Strooper, “Animation can show only the presence of errors, never their absence,” in Proceedings of 2001 Australian Software Engineering Conference, IEEE CS Press, 2001, pp. 76–88. [8] A. Aurum, H. Petersson, and C. Wohlin, “State-of-the-art: Software inspections after 25 years,” Software Testing, Verification and Reliability, vol. 12, no. 3, pp. 133–154, 2002. [9] N. G. Leveson, Safeware Engineering: Engineering for a Safe World. : [Online]. Available: http://www.safeware-eng.com/index.php/whitepapers/verification, Webpage article [10] A. Hall, “Realizing the benefits of formal methods,” in Proceedings of 7th International Conference on Formal Engineering Methods, LNCS 3785, Nov. 1–4, 2005, pp. 1–4, Springer-Verlag. [11] M. Chechik and J. D. Gannon, “Automatic analysis of consistency between requirements and designs,” IEEE Trans. Software Engineering, vol. 27, no. 7, pp. 651–672, 2001. [12] K. McMillan, Symbolic Model Checking: An Approach to the State Explosion Problem. : Kluwer Academic Publisher, 1993. [13] Z. Duan, C. Tian, and L. Zhang, “A decision procedure for propositional projection temporal logic with infinite models,” Acta Informatica, no. 45, pp. 43–78, 2008. [14] M. Hewitt, C. O’Halloran, and C. Sennett, “Experiences with PiZA: an Animator for Z,” in Proceedings of the 1997 Z User Meeting (ZUM’97), LNCS 1212, 1997, pp. 37–51, Springer-Verlag. [15] H. Waeselynck and S. Behnia, “B Model Animation for External Verification,” in Proceedings of the Second IEEE International Conference on Formal Engineering Methods (ICFEM 98), 1998, pp. 36–45, IEEE CS Press. [16] M. E. Fagan, “Design and code inspections to reduce errors in program development,” IBM Systems Journal, vol. 15, no. 3, pp. 182–211, 1976. [17] J. C. Knight and E. A. Myers, “An improved inspection technique,” Communications of the ACM, vol. 36, no. 11, pp. 51–61, 1993. [18] 1028-1997 {IEEE} Standard for Software Reviews, 1997, IEEE CS Press, IEEE. [19] D. L. Parnas and D. M. Weiss, “Active design reviews: principles and practices,” in Proceedings of the 8th International Conference on Software Engineering, August 1985, pp. 215–222. [20] T. Heyer, “Semantic inspection of early UML designs,” IEEE Trans. Software Engineering, vol. 28, no. 4, pp. 413–430, 2002. [21] S. Liu, Formal Engineering for Industrial Software Development Using the SOFL Method. : Springer-Verlag, 2004. [22] S. Liu, A Case Study of Modeling an ATM Using SOFL Hosei University, Tokyo, Japan, Technical Report HCIS-2003-01, 2003. [23] E. Yourdon, Modern Structured Analysis. : Prentice Hall, 1989. [24] C. B. Jones, Systematic Software Development Using VDM, 2nd ed. : Prentice Hall, 1990. [25] W. Brauer, G. Rozenberg, and A. Salomaa, Petri Nets: An Introduction. Berlin Heidelberg: Springer-Verlag, 1985. [26] C. Ho-Stuart and S. Liu, “A formal operational semantics for SOFL,” in Proceedings of the 1997 Asia-Pacific Software Engineering Conference, Hong Kong, Dec. 1997, pp. 52–61, IEEE CS Press. [27] S. Liu, A. J. Offutt, C. Ho-Stuart, Y. Sun, and M. Ohba, “SOFL: A formal engineering methodology for industrial applications,” IEEE Trans. Software Engineering, vol. 24, no. 1, pp. 337–344, January 1998, IEEE CS Press. [28] S. Liu, “Developing quality software systems using the SOFL formal engineering method,” in Proceedings of the 4th International Conference on Formal Engineering Methods (ICFEM 2002), Shanghai, China, Oct. 21–25, 2002, pp. 3–19, LNCS 2495, Springer-Verlag. [29] S. Liu, M. Asuka, K. Komaya, and Y. Nakamura, “An approach to specifying and verifying safety-critical systems with practical formal method SOFL,” in Proceedings of the Fourth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS’98), California, USA, August 10–14, 1998, pp. 100–114, IEEE CS Press, Monterey.

683

[30] H. Chen, Y. Shen, and J. Jiang, “Extended SOFL features for the modeling of middleware-based transaction management,” in Proceedings of the 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS2005), Presented at the Workshop on SOFL, Shanghai, China, 2005, pp. 16–17, IEEE CS Press. [31] S. Liu, M. Shibata, and R. Sat, “Applying SOFL to develop a university information system,” in Proceedings of 1999 Asia-Pacific Software Engineering Conference (APSEC’99), December 6–10, 1999, pp. 404–411, IEEE CS Press. [32] J.-R. Abrial, The B-Book: Assigning Programs to Meanings. : Cambridge University Press, 1996. [33] J. Warmer and A. Kleppe, The Object Constraint Language: Getting Your Models Ready for MDA. : Addison-Wesley, 2003. [34] Y. Chen, S. Liu, and F. Nagoya, “An approach to integration testing based on data flow specifications,” in Proceedings of First International Colloquium on Theoretical Aspects of Computing, Guiyiang, China, September 20–24, 2004, pp. 235–249, LNCS, Springer-Verlag. [35] A. Bloesch, E. Kazmierczak, P. Kearney, and O. Traynor, “Cogito: Methodology and system for formal software development,” International Journal of Software Engineering and Knowledge Engineering, vol. 5, no. 4, pp. 599–618, 1995. [36] E. Kazmierczak, P. W. Dart, L. Sterling, and M. Winikoff, “Verifying requirements through mathematical modeling and animation,” International Journal of Software Engineering and Knowledge Engineering, vol. 10, no. 2, pp. 409–439, 2000. [37] T. Miller and P. Strooper, “A framework and tool support for the systematic testing of model-based specifications,” ACM Trans. Software Engineering and Methodology, vol. 12, no. 4, pp. 409–439, 2003. [38] M. S. Jaffe, N. G. Leveson, M. P. E. Heimdahl, and B. E. Melhart, “Software requirements analysis for real-time process-control systems,” IEEE Trans. Software Engineering, vol. 17, no. 3, pp. 241–258, March 1991. [39] W. E. Vesely, F. F. Goldberg, N. H. Roberts, and D. F. Haasl, Fault Tree Handbook. Washington, D.C. 20555: NUREG-0492, Systems and Reliability Research Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission, January 1981. [40] T. Gilb and D. Graham, Software Inspection. : Addison-Wesley, 1993. [41] S. Liu, “Verifying consistency and validity of formal specifications by testing,” in Proceedings of the World Congress on Formal Methods in the Development of Computing Systems, J. M. Wing, J. Woodcock, and J. Davies, Eds., Toulouse, France, September 1999, pp. 896–914, LNCS, Springer-Verlag. [42] F. MacDonald and J. Miller, “A comparison of tool-based and paperbased software inspection,” Empirical Software Engineering, vol. 3, no. 3, pp. 233–253, 1998. [43] J. S. Fitzgerald, P. G. Larsen, P. Mukherjee, N. Plat, and M. Verhoef, Validated Designs for Object-oriented Systems. : Springer-Verlag, 2005. [44] VDMTools User Manual (VDM-SL) ver. 1.0.. Tokyo, Japan: CSK SYSTEMS CORP.. [45] S. Owre, N. Shankar, J. M. Rushby, and D. W. J. Stringer-Calvert, PVS System Guide ver. 2.4. , USA: SRI, 2001. [46] N. G. Leveson and P. R. Harvey, “Analyzing software safety,” IEEE Trans. Software Engineering, vol. 9, no. 5, pp. 569–579, September 1983, IEEE CS Press. [47] N. G. Leveson, Safeware: System Safety and Computers. : AddisonWesley, 1995. [48] D. L. Parnas and D. M. Weiss, “Active design reviews: Principles and practice,” in 8th International Conference on Software Engineering, 1985, pp. 132–136. [49] K. E. Wiegers, Peer Reviews in Software: A Practical Guide. : Addison-Wesley, 2001. [50] A. A. Porter, H. P. Siy, and L. G. Votta, “A review of software inspections,” Advances in Computers, vol. 42, pp. 39–76, 1996. [51] Software Formal Inspections Standard, NASA-STD-2202-93, 1993, NASA. [52] B. Cheng and R. Jeffrey, “Comparing inspection strategies for software requirements specifications,” in Proceedings of the 1996 Australian Software Engineering Conference, 1996, pp. 203–211. [53] V. R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shulland, S. Sorumgard, and M. Zelkowitz, “The empirical investigation of perspective-based reading,” Empirical Software Engineering, vol. 2, no. 1, pp. 133–164, 1996.

684


[54] A. A. Porter, L. G. Votta, and V. R. Basili, “Comparing detection methods for software requirements inspections: A replicated experiment,” IEEE Trans. Software Engineering, vol. 21, no. 6, pp. 563–575, 1995. [55] A. A. Porter and L. G. Votta, “An experiment to assess different defect detection methods for software requirements inspections,” in Proceedings of the 16th International Conference on Software Engineering, Sorrento, Italy, Aug. 1994, pp. 103–112. [56] T. Thelin, P. Runeson, and C. Wohlin, “An experimental comparison of usage-based and checklist-based reading,” IEEE Trans. Software Engineering, vol. 29, no. 8, pp. 687–704, August 2003. [57] O. Laitenberger and C. Atkinson, “Generalising perspective-based inspection to handle object-oriented development artifacts,” in 21st International Conference on Software Engineering, 1999, pp. 494–503. [58] P. Anderson, T. Reps, and T. Teitelbaum, “Design and implementation of a fine-grained software inspection tool,” IEEE Trans. Software Engineering, vol. 29, no. 8, pp. 721–733, August 2003. [59] H. D. Mills, M. Dyer, and R. Linger, “Cleanroom software engineering,” IEEE Software, vol. 4, no. 5, pp. 19–25, September 1987. [60] A. Dunsmore, M. Roper, and M. Wood, “The development and evaluation of three diverse techniques for object-oriented code inspection,” IEEE Trans. Software Engineering, vol. 29, no. 8, pp. 677–686, August 2003. [61] C. L. Heitmeyer, R. D. Jeffords, and B. G. Labaw, “Automated consistency checking of requirements specifications,” ACM Trans. Software Engineering and Methodology, vol. 5, no. 3, pp. 231–261, 1996. [62] N. G. Leveson, J. D. Reese, and M. P. E. Heimdahl, “SpecTRM: A CAD system for digital automation,” in Proceedings of 17th Digital Avionics System Conference (DASC 98), Seattle, USA, November 1998. [63] Z/EVES [Online]. Available: http://www.ora.on.ca/z-eves/ ORA Canada, Z/EVES homepage [64] J. Woodcock, P. G. Larsen, J. Bicarregui, and J. Fitzgerald, “Formal methods: Practice and experience,” ACM Computing Surveys, vol. 41, no. 4, 2009.

Shaoying Liu (M’94–SM’07) is Professor of Software Engineering at Hosei University, Japan. He received a Ph.D. in Computer Science from the University of Manchester, U.K. in 1992. His research interests include Formal Engineering Methods for Software Development, Specification Verification and Validation, Specification-based Program Inspection, Specification-based Program Testing, and Intelligent Software Engineering Environments. He has published a book titled “Formal Engineering for Industrial Software Development Using the SOFL Method” with Springer-Verlag, four edited conference proceedings, and over 110 academic papers in refereed journals and international conferences. He proposed to use the terminology of “Formal Engineering Methods” in 1997, has established Formal Engineering Methods as a research area based on his extensive research on the SOFL (Structured Object-oriented Formal Language) method since 1989, and the development of ICFEM conference series since 1997. In recent years, he has served as Program Co-Chair of International Conference on Formal Engineering Methods (ICFEM 2008), the Chair of the Steering Committee for ICFEM 2010, and PC member for many international conferences. He is on the editorial board for the Journal of Software Testing, Verification and Reliability (STVR). He is a Fellow of British Computer Society, a Senior Member of IEEE Computer Society, and a member of Japan Society for Software Science and Technology.

John A. McDermid has been Professor of Software Engineering at the University of York since 1987 where he runs the High Integrity Systems Engineering (HISE) research group. HISE studies a broad range of issues in systems, software, and safety engineering; and works closely with government and industry, e.g. Airbus, BAE Systems, the Civil Aviation Authority, QinetiQ and Rolls-Royce. He is author or editor of six books and has published about 330 papers. He has advised companies and government departments on several continents, most recently advising the US Nuclear Regulatory Commission (NRC) on software safety. He has given tutorials to academic and industrial audiences on five continents. He was elected a Fellow of the Royal Academy of Engineering in 2002, and was made an Officer of the Order of the British Empire (OBE) in the 2010 New Year’s Honours list.

Yuting Chen received a Ph.D. in Computer Science from Hosei University, Japan in 2007. He is currently an Assistant Professor in the School of Software, Shanghai Jiaotong University, China. His research interests include software engineering, applied formal methods, and software verification and validation.

A Rigorous Method for Inspection of Model-Based ... - Semantic Scholar

A Rigorous Method for Inspection of Model-Based ... - Semantic Scholar

Suggest Documents

Modelbased analysis of skin conductance ... - Semantic Scholar

A Rigorous, Operational Formalization of ... - Semantic Scholar

A Method for Teaching Rigorous Mathematical Reasoning

a rigorous review and thorough planning for the ship inspection ...

A RIGOROUS RENORMALIZATION GROUP ... - Semantic Scholar

Implications for Rigorous Glycemic Control - Semantic Scholar

A Rigorous Optimization Method of Operating ...

A modelbased PID controller for Hammerstein

Using a Rigorous Approach for Engineering Web ... - Semantic Scholar

KELVIN: A Software Package for Rigorous ... - Semantic Scholar

KELVIN: A Software Package for Rigorous ... - Semantic Scholar

A Rigorous Optimization Method of Operating Parameters for Amine

Rigorous Version of Infinitesimal Deformation ... - Semantic Scholar

CoverageAnalyzer (CAn): A Tool for Inspection of ... - Semantic Scholar

On the Advantages of a Systematic Inspection for ... - Semantic Scholar

Semiotic Inspection of a game for children with ... - Semantic Scholar

SPIDAR: SPinneret for Inspection And ... - Semantic Scholar

a rigorous determination of the overall period in ... - Semantic Scholar

A Rigorous Extension of the SchÂ¨onhage-Strassen ... - Semantic Scholar

A Rigorous Evaluation of Crossover and Mutation ... - Semantic Scholar

Modelbased analysis of skin conductance

A rigorous unidirectional method for designing finite ... - OSA Publishing

a rigorous and flexible calibration method for digital airborne camera ...

A Rigorous Model of Service Component ... - Semantic Scholar