An Approach to Detecting Domain Errors Using Formal Specification ...

An Approach to Detecting Domain Errors Using Formal Specification-Based Testing∗ Yuting CHEN, Shaoying LIU Faculty of Computer and Information Sciences Hosei University, Tokyo, Japan E-mail: {i0t49001, sliu}@k.hosei.ac.jp

Abstract

the predicate faults in conditional statements, or from assignment faults in program[9]. A domain error can be manifested by a shift or a tilt in some segment (border) of the path domain boundary, and therefore incorrect output is generated due to executing a wrong path through the program[10]. In this paper, we introduce a strong domain testing strategy based on formal specifications. Domain testing, which was proposed as a relatively sophisticated form of boundary value testing, is used to test software or portions of software dominated by numerical processing[2]. Domain testing is applicable whenever the input domain is divided into sub domains by the program’s decision statements. Several domain testing strategies have been proposed to catch domain errors[9][10][11][12][13][14], and our strong strategy is different in the following two aspects:

Domain testing, a technique for testing software or portions of software dominated by numerical processing, is intended to detect domain errors that usually arise from incorrect implementations of designed domains. This paper describes our on-going work aiming to provide support for revealing domain errors using formal specifications. In our approach, formal specifications serve as a means for domain modeling. We describe a strong domain testing strategy that guide testers to select a set of test points so that the potential domain errors can be effectively detected, and apply our approach in two case studies for test cases generation. Keywords: domain errors, domain testing, formal specification, SOFL, test point selection

1

1. the test points are to check the consistency of a program domain with its specification domain, that is, we select a set of test points from a specification domain and check whether they belong to its program domain;

Introduction

Specification-based software testing is a way to enhance the reliability of software systems[1][2][3][4]. It advocates the idea that test cases are generated based on the functional specifications of systems. Recent approaches[5][6][7][8] to specification-based testing are mostly concerned with formal specifications. A formal specification defines a system and its desired properties using a language with a mathematically-defined syntax and semantics, thus it provides a significant opportunity for rigorous functional testing. This paper describes our on-going work aiming to provide support for revealing domain errors using formal specifications. A classification of program errors with strong intuitive appeal is the division into domain errors and computation errors, in which domain errors usually arise from

A test point is a set of values of all of the input variables consisting of one value for each variable. In our approach, the test points are selected not from an executable program but from a formal specification. Since a specification serves as the basis for a “contract” or mutually agreed expectation between customer and supplier, and rigorous refinement requires that a program should be consistent with its specification, the test points derived from the formal specification to reflect the user requirements may serve as the “use cases”. Besides, in order to derive a program domain, the testers need to make a great effort to analyze the program and its conditional structure. Because a formal specification defines a system at an abstract level, the cost of domain modeling and test point selection will likely be decreased.

∗ This work is supported by the Ministry of Education, Culture, Sports, Science and Technology of Japan under Grant-in-Aid for Scientific Research on Priority Areas (No.15017280).

2. the selected test points can effectively cover the given domain and reveal most domain errors. 1

Domain errors usually include: closure bug, border shift “up” or “down”, border tilt, extra boundary, missing boundary[2]. Most domain testing strategies are concerned with the issue of how to reveal closure bugs, border tilt errors, and border shift errors, but are blind to the extra boundary. These strategies are probably good enough for testing ordinary software, but for life critical software, or for software that creates considerable financial exposure, one might want to play it safe and use stronger methods to guarantee that sneaky bugs will not be unnoticed. Our strategy has an ability to cover a specification domain and to discover all of the above mentioned domain errors. The remainder of this paper is organized as follows. Section 2 and 3 briefly introduce the related work and a formal specification viewpoint of domain errors, respectively. Section 4 presents how to select test points from a specification domain. Section 5 introduces our objective specification language-SOFL and shows two case studies of generating test points from formal specifications. Finally in Section 6, we give conclusions and point out future research.

2

Clarke et al. [11] showed that certain domain errors went undetected by the White and Cohen strategy, and proposed two strategies intended to improve domain testing. If a given border contains V vertices, their V × V strategy requires V ON points selected as close as possible to each vertex, and V OFF points chosen at a uniform distance from the given border. Their V ×1 strategy requires V ON points and one OFF point. [14] described a simplified version of domain testing that is applicable to arbitrary types of predicates and detects errors from domains with both linear and nonlinear boundaries. In addition, an approach to path selection was used in conjunction with the simplified domain testing strategy. Few commercial tools implement proper domain testing, since the early domain testing strategies have several drawbacks to limit their application in industry[2][14]. The techniques of domain modeling and test point selection from practical programs are still immature because of the complexity of programs. Furthermore, domain errors can be broken up further into two classes: path selection errors and missing path errors. A path selection error happens when both the correct paths and the wrong paths exist in the program, and a wrong path is selected and executed by any of the test points. A missing path error means a required predicate does not appear in the given program to be tested. Most domain testing strategies based on programs lack the ability to detect the latter.

Related Work

A path domain of a program is surrounded by a boundary. A boundary can usually be divided into borders, each of which corresponds to a predicate interpretation of the path condition[9]. A border may either be closed or opened with respect to that domain. A closed border belongs to the path domain and comes from a predicate that contains ≤, ≥, or =. An open border is not part of the path domain and comes from a predicate that contains , or =. The point at which the borders intersect is a vertex. Domain testing was initially proposed by White and Cohen[9][10], who indicated that points near the boundary of a path domain are more sensitive to program faults than other points, and then developed systematic strategies aimed at revealing domain errors within some error bound. The first domain testing strategy of White and Cohen, known as the N × 1 strategy, begins with a path that has been chosen from program by some path selection method. Assume the given border of an N −dimensional linear case is closed and represents an inequality predicate interpretation, N × 1 strategy requires two types of points to be selected as test cases: ON points lie on the given border, and OFF points lie slightly off the given border on the open side of it. The N × 1 strategy uses N ON points to identify the given border, and one OFF point to ensure that the correct border does not lie on the open side of the given border. When we test an open border, the roles of the ON and OFF points are reversed. The N × 1 domain-testing technique was for inequality borders. If the given border is an equality constraint, White and Cohen proposed a N × 3 strategy, which requires N ON points and three OFF points.

Domain testing strategies are also widely applied in specification-based testing. [5] applied formal methods to software testing, in which domain testing is integrated in a test template framework to generate tests from Z specifications. [15][16] and [17] describe domain testing using Boolean specifications and character string predicates, respectively. However, these studies are mostly concerned with unit testing, but not with integration testing, which is necessary for large-scale systems development in industry. Secondly, we believe that the early domain testing strategies should not be directly adopted to specification-based testing, because they were proposed to select test points from a hypothetically incorrect program domain, while during specification-based testing, the specification domains are usually supposed to be consistent and valid. In addition, bugs are more insidious than ever we expect them to be[12]. Our study shows that domain errors exist not only around the border, but also probably everywhere. To address this problem, we need more powerful domain testing strategies for selecting test points based on formal specifications, which is especially necessary and important for life critical systems. 2

3

A Specification Viewpoint of Domain Errors

flow diagram; a predicate in a Z specification; a precondition and a postcondition in a VDM specification; a procedure, a statement block, or a decision in a structural program; or a method in an object oriented program. Thus a domain error caused by the violation of the consistency between a program domain and its specification domain induces the wrong paths of the program to be taken during execution. The relations between a P D and a SD of the same function can be categorized as

Traditionally, a domain error can be manifested by a missing boundary, an extra boundary, a wrong closure, a shift or a tilt in some segment (border) of the path domain boundary[2], as Figure 1 shows, and therefore incorrect output is generated due to executing wrong paths[10]. From the viewpoint of a specification, it is much easy to rigorously define a domain error by comparing a specification domain and its program domain. B

Incorrect

A

Correct

(a) Cl osur e Bug

Incorr ect Corr ect (b) Bor der Shi f t Er ror

Disjoint : P D ∩ SD = ∅

Case (2)

Equivalence : P D = SD

Case (3)

Subset : P D

Case (4)

Incorrect

I ncor rect

Correct

Corr ect

(c) Bor der Ti l t Err or

Case (1)

Case (5)

⊂ SD Superset : P D ⊃ SD Overlap : SD − P D =  ∅ and P D − SD =  ∅ and P D ∩ SD = ∅

These relations are illustrated by the Venn diagram in Figure 2, where domain errors exist in Cases (1), (3) and (5). In the next section, we will introduce a domain testing strategy to detect domain errors from a formal specification domain. This strategy is stronger than the N × 1 strategy, N × 3 strategy, and V × V strategy in that it has the same effectiveness as the early strategies in revealing closure bugs, border shift errors, and border tilt errors, but it has an ability to detect the extra boundary which the previous strategies are blind to.

(d) Ext r a Boundary Cor rect

(e) Mi ssi ng Boundary Exampl e: Corr ect Border Incorrect Border Cl osur e Bug x