Contract-Based Mutation for Testing Components - CiteSeerX

Contract-Based Mutation for Testing Components Ying Jiang1,2, Shan-Shan Hou1, Jin-Hui Shan1, Lu Zhang1, Bing Xie1 1(Institute of Software, School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, China) 2(Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, 650093, China) {jiangy, houss, shanjh, zhanglu, xiebing}@sei.pku.edu.cn

Abstract Testing plays an important role in the maintenance of Component Based Software Development. Test adequacy for component testing is one of the hardest issues for component testing. To tackle this problem, it is a natural idea to apply mutation testing, which is a fault-based testing method used for measuring test adequacy, for component contracts, whose aim is to improve the testability of the component. Though powerful, mutation testing is usually very computationexpensive, as many mutants need to be produced and executed in mutation testing. In this paper, we propose a contract-based mutation technique for testing components. Based on the discordance between contracts and specification, our approach employs a set of high level contract mutation operators. The experimental results show that these operators can greatly reduce the number of mutants compared with traditional mutation operators. At the same time, the contract-based mutation using our contract mutation operators can provide almost the same ability as that of using traditional mutation operators. Moreover, effective test suite can be produced to reduce the maintenance effort.

1. Introduction With the advent of Component Based Software Development (CBSD), more and more software systems are assembled from reusable components. Research has shown that reuse may improve both software quality and development productivity [10, 24]. Naturally, the success of CBSD highly depends on the quality of reusable components used in the development process. In order to make the components reusable and reliable, new testing methods are needed as testing components may be quite different from

testing traditional software [33]. Ideally, the developer of a component should test it in different application domains. However, as developer cannot assume all kinds of future reuse contexts of the component, so users usually need to test components again in their application environment. Compared with the research on CBSD, only a limited number of efforts have addressed the problems in testing reusable software components and component-based software [3, 5, 12, 14, 17, 37]. Gao et al listed component testing issues and challenges in the perspectives of component users and vendors [11]. Among the issues in component testing, test adequacy is one of the toughest issues. Although many test criteria have been proposed and studied in order to measure the quality of traditional software testing, most of them are code-based and cannot be used for components whose source code is unavailable. Furthermore, although some existing black-box based criteria are available to software components, new features introduced by component-based software, such as component reuse, customization, composition and deployment, may threaten the validity of these criteria [11]. Usually, component developers and users have different knowledge, understanding and visibility of the component. The developers have the whole source code of the component whereas the users frequently cannot gain all information of the component. So they may use different testing methods and test criteria. It is still a problem what criteria can be applied to evaluate the component testing by both component developers and users. Mutation testing, which is introduced by Hamlet [15] and DeMillo et al [4, 6], is a widely accepted adequacy criterion to measure test quality. However, the huge computational cost caused by executing thousands of mutants and detecting plenty of equivalent mutants manually is an obstacle of applying mutation testing in practice.

Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM’05) 1063-6773/05 $20.00 © 2005 IEEE

In this paper, we propose a contract-based mutation technique that may serve as a test adequacy criterion for components. The basic idea of our approach is to apply mutation testing to the contracts provided along with components. As the contracts can be supplied without source code, mutating the contracts does not depend on the source code of components. In order to overcome the drawback of requiring large computation resources, our approach does not use the traditional mutation operators for contracts [1, 18]. Instead, we mutate component contracts at a higher level. Some experiments on contract mutation are performed, and the experimental results show 1) that the contract mutation score can be used to evaluate the test data as a test adequacy criterion just like the statement coverage score and branch coverage score; and 2) that contractbased mutation using the mutation operators proposed in our approach can be more efficient than using the traditional mutation operators. Moreover, the test suite selected by contract-based mutation can be reused during component regression testing, which can reduce the cost of regression testing. The remainder of this paper is structured as follows. In section 2, we describe the background knowledge of contracts. Differing from the traditional mutation, we present a new contract-based mutation in Section 3. Section 4 contains the description and analysis of our experiments. In Section 5, related work is discussed. A summary, some conclusions and plans for future work appear in Section 6.

natural idea to add contracts to components to improve their testability. For a software system composed of a collection of components, these components interact with each other through their interfaces. Without contracts of each component, some faults may not be revealed by contract-breaking executions easily. It is even harder to trace which component should take the responsibility when a fault is revealed. The component contracts can define the required condition of using the component and the function implemented by the component. Thus, these contracts can help to identify whether the component should take the responsibility. Besides the benefit of enhancing the testability of a component, component contracts can also help the implementation of the component. In fact, the software specification can be partially specified by the contracts. Thus, software implementation can be much easier and more efficient with the guidance of the contracts, because the provider does not need to check the use condition of the software in the source code explicitly. In [31], Szyperski pointed out that contracts can make reusable building blocks easier to implement, test and compose. In this paper, our approach is aiming at mutating component contracts. As component contracts have the above benefits in component-based software development, it should not be a burden for an approach to require the co-existence of component contracts with the component.

3. Contract-based mutation 2. Background Design by Contract (DbC) is a method to improve software testability. Increasing software testability means designing the software in a certain way that faults are more likely to reveal themselves if the software is faulty. Once software testability is increased, the efficiency of software testing can be enhanced while the cost is reduced. The basic idea of DbC is to establish the contracts between the provider and the user of a software entity [23]. These contracts can help to distinguish the responsibility of the provider from that of the user. Because contracts describe the behavioral features of components, whether contracts are violated can be known when the components are running. During testing, any execution that breaks one or more contracts can be viewed as a fault-revealing execution, and the broken contract(s) can be used to trace the fault. Typically, the contracts include preconditions, postconditions, class invariants, loop variants, and loop invariants, etc. Although DbC is not originally proposed for component-based software development, it should be a

In this section, we will present our approach in detail. The mutated contracts can represent programmers’ possible miscomprehension of requirements, or implementation errors in terms of contracts. Like any other approach to mutation testing, our approach uses mutation operators to generate mutants, and uses a mutation score that defines the ability of killing mutants to measure the quality of test suites. The distinctive features of our approach are as follows. Firstly, our approach aims at mutating component contracts defined in section 3.1. Secondly, our approach employs a set of high level mutation operators, whose main purpose is to generate fewer mutants without incurring intolerable information loss. These operators are presented in section 3.2. Thirdly, as the target of our mutation is not source code, the oracle for source code-based mutation is not applicable for our approach. The definition of our mutation oracle is presented in section 3.3. Finally, we present the definition of the mutation score for our approach in section 3.4. Our definition is almost identical to the

Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM’05) 1063-6773/05 $20.00 © 2005 IEEE

traditional definition except that our definition is based our mutation oracle.

3.1. Component contracts 3.1.1. Definition. OCL [29] and JML [20] provide the description about contracts. However, OCL cannot describe generality quantifier and existential quantifier of contracts. JML is too complex for component interfaces. Therefore, we define a simple language to describe the interface contracts through extending the definition of Enterprise Java Bean (EJB) components (please refer to http://cui.unige.ch/dbresearch/Enseignement/analyseinfo/JAVAF/BNFindex. html for the BNF rules for Java). Its aim is to test EJB components. The grammar of this language is listed as follows. component ::= Component comp-name interface-section End Component. interface-section ::= Interface method-list End Interface. method-list ::= method { ; method } ; method ::= method-name [ ( para-list ) ] returns type_specifier contract-spec contract-spec ::= /* {PreconditionSection} {PostconditionSection} */ PreconditionSection ::= @pre [pre-name:] ContractExpression ; PostconditionSection ::= @post [post-name:] ContractExpression ; ContractExpression ::= Expression | forall Type() VariableDeclarator() in collection Expression | exists Type() VariableDeclarator() in collection Expression | Expression implies Expression Expression ::= numeric_expression | testing_expression | logical_expression | string_expression | casting_expression | literal_expression | null | super | this | Old | Result | identifier | ( ( expression ) ) | ( expression ( [ arglist ] ) ) | ( [ expression ] ) | ( . expression ) collection ::=

{ element { , element } } element ::= integer_literal … integer_literal | character … character | integer_literal | float_literal | character | string

In the above grammar, a precondition expresses the conditions under which the component interface will function properly. A postcondition expresses properties of the results when a component interface has executed correctly. The special notations Old and Result can only be used in postconditions. The Result means the return value of an interface. The notation Old.e, where e is an expression, denotes the value that e had on interface entry. Any occurrence of e not preceded by Old in the postcondition denotes the value of the expression on interface exit. 3.1.2. An example. Supposing that a withdraw operation of an Automated Teller Machine (ATM) component is requested and the relevant interfaces are ValidatePin and WithDraw, the corresponding contracts of the two interfaces are listed as follows: public interface atmRemote extends EJBObject { public int ValidatePin(String inputID,String inputPin) throws RemoteException; / * @pre inputID.length()==4; @pre inputPin.length()==4; */ public boolean IsValid() throws RemoteException; public int Query() throws RemoteException; public int WithDraw(int inputAmount) throws RemoteException; / * @pre (inputAmount>0) && (inputAmount=Q P
= Q P,== ,< P>Q+constant P=,=,!= =Q-constant P