Regression Testing for Component-based Software via Built-in Test Design Chengying Mao1,2,3 1. College of Comp. Scie. & Tech., Huazhong Univ. of Scie. & Tech. Wuhan, Hubei 430074, China
Yansheng Lu1
Jinlong Zhang3
2. School of Software, Jiangxi Univ. of Finance & Economics Nanchang, Jiangxi 330013, China
3. College of Management, Huazhong Univ. of Scie. & Tech. Wuhan, Hubei 430074, China
[email protected]
D.2.5 [Software Engineering]: Testing and Debugging – Testing tools; D.2.4 [Software Engineering]: Software/Program Verification – Validation
system or remove bugs. After each system modification, it is quite necessary to retest the system to provide confidence that the modifications are correct and have not adversely affected other parts of the system, that is so-called regression testing. While performing regression testing, it is unnecessary to rerun all existing test cases but to pick out the subset which can cover the potentially-affected code of program. Since component user has no knowledge of detailed internal constructs and change information about component that is not internally developed, it is fairly hard to tackle this kind of change in component-based software. The ultimate reason of the trouble in regression testing for CBS, which is built by some externally-developed components, is the lack of information about those components and their changes. Tester neither knows the internal structure information of component nor the change information and its impact. The rational solution is to enhance the information exchange between component developer and user [1]. Through analyzing the shortcomings of existing regression testing techniques for CBS, we presented an improved regression testing method, i.e., regression testing based on component built-in test design. Furthermore, the preliminary experiment about the proposed method is addressed, and experimental results show that our method is feasible and cost-effective in practice.
General Terms
2. BUILT-IN TEST DESIGN
Algorithms, Design, Reliability, Verification
Component user is often confronted with the problem that the necessary information for testing purposes is unavailable. In fact, the underlying cause for this problem is the weak testability of component. It is necessary to merge the idea and principle of introducing testing into the phases of software design and coding with the precondition of little or even no increase of software complexity, so as to improve the controllability and observability of component in testing process. At present, quite a few component’s testability improvement methods have been proposed, such as intro-/retro-spection approach [2], wrapperbased method [1,3,4], component meta-data approach [5] and so on. The most representative one of them is the built-in test approach, which has been popularly utilized for testing the single component or component-based system. The basic idea of built-in test design is that component provider pre-places test scripts in component and sets the corresponding testing-interfaces. In general, the built-in test scripts contain test cases or can posses facilities capable of generating the test cases which the component can be use to test itself and its own methods [6]. A component can operate in two modes, namely in normal
ABSTRACT Component-based software technology is expected to be an effective and widely used method of constructing software system. However, some specialties of component bring a great challenge for testing the systems built by externally-provided components, especially for regression testing. Built-in test design is a fairly effective way to improve component’s testability. In this paper, we present an improved regression testing method based on builtin test design for component-based systems. It needs the mutual collaboration between the component developers and users. Component developers are responsible for analyzing the affected methods and constructing the corresponding testing-interfaces in the new component version, and then component users can conveniently pick out the subset of test cases for regression testing with these testing-interfaces. Through employing preliminary experiments on some medium scale systems, our regression testing method based on built-in test design has been proven to be fairly feasible and cost-effective in practice.
Categories and Subject Descriptors
Keywords Component, Regression testing, Method call graph, Built-in test design, Test case selection
1. INTRODUCTION During the evolution of component-based software system (CBS for short), portions of the software may be removed, replaced or added in order to improve the functionality of the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC’07, March 11-15, 2007, Seoul, Korea. Copyright 2007 ACM 1-59593-480-4/07/0003…$5.00.
1416
are denoted as published variables (PVs), correspondingly, the set of PVs is Dp = Dw ∪ Dr . M = ( Mp, Mh) is the set of component’s methods (a.k.a. functions or operations). Mp is the set of published methods (PMs), while Mh is the set of methods which can’t be called by external user. Generally, several published variables and published methods constitute an interface l ∈ L (here L is an interface set of component), i.e., l = {mpi, ...} ∪ {dwj , ...} ∪ {drk , ...}(mpi ∈ Mp, dwj ∈ Dw, drk ∈ Dr ) .
mode and maintenance mode [7]. While running in the normal mode, the component doesn’t differ from other, non-built-in testing enabled components. However, in the maintenance mode, the component user can invoke the respective methods of component to execute the test, evaluate the test results and output a test report. The underlying principle can be schematically demonstrated in the following figures, Figure 1 is a typical framework of built-in test component (test suit is optional). Functional Methods
A component is usually realized by one or more classes, but its information encapsulation is much stricter than class, and external program can merely access the published variables (PVs) and published methods (PMs). In most component models, the interface information provided to the outer users is generally organized in a standard format (e.g., interface description language, IDL). With the help of application developer’s program code, some components can be assembled or integrated into a system. We also analyze the component-base system from the perspective of composition, and define it as below.
Normal Interface Normal Scripts Testing Interface
Test Suit
Testing Scripts Testing Methods
Figure 1. The framework of built-in test component.
Definition 2 CBS is also defined as a 2-tuple S = ( P , SC ) , where: P is system developer’s own code which is used to assemble components into an application. Without loss of generality, it can be viewed as a set of methods, i.e., P = {m1, ..., mn} . SC is the set of components used in this component-based software system, i.e., SC = {C1, ..., Cm} .
Built-in test design exhibits the merits as follows: The component providers can design rational test framework and costeffective test cases on the condition of their sufficient comprehension of component’s internal structure, so component users can save plentiful test efforts. Built-in test approach is usually adopted to test the component’s compatibility for real execution context [8], or used for the integration testing of component-based software system [9,10]. In fact, this technology can also be adjusted and then employed to regression testing for CBSs. Here, our purpose is not to test a single component, its methods, or its interaction with other components, but only to get some association information about system’s test cases and the change in a component via built-in test design. Therefore, in order to facilitate system’s regression testing, it is unnecessary to embed test cases or scripts which can generate test cases into component, but to seed the scripts which can gather the execution information (i.e., whether a test case walks through the changed statements or not into the new component version).
High evolvability is a remarkable character of componentbased system. Code modification in component’s method is a familiar and common change in CBSs. It is fairly difficult to explore the faults caused by this change, and they can usually be found only at run-time. In this paper, we mainly discuss this situation, i.e., some code changes occur in a method of component. To facilitate the discussion in the following sections, some terms are defined as below. Definition 3 the changed statement within some method is called as change point, and CP for brevity. Definition 4 the methods are directly or indirectly affected by a CP, we call them affected methods with respect to that CP. Furthermore, the set of affected methods is named change information (i.e., CI).
3. BASIC KNOWLEDGE 3.1 Component and Its Changes To date, a uniform and standard definition has not been achieved yet [11]. In this paper, we define component as a nearly independent, replaceable module which encapsulates data and operations used to implement some specific functions. Component can be constructed by many programming technologies. Here, we assume that component is developed by the Object-Oriented design pattern and programming languages, such as Java, C++/C# and so on. In view of the constitutive character of component, we can define it as follows.
Since components generally use methods as building blocks for services, the strategy of considering modification at method level is reasonable and feasible [12]. The outer application usually calls the component’s published methods via using the statement like objC. foo(var1, var 2,..., vark ) , here objC is instantiated object of the component C . As stated in definition 2, the outer application can also be viewed as a set of n methods, so the execution of the whole component-based system under each input can be denoted as a path composed of a series of application’s methods. In this paper, in order to facilitate the discussion, we assume the test case is expressed as method-call sequence, where fi (...) ∈ P = i.e., tc = f 1(...), ..., fi (...), ..., ft (...) , {m1, ..., mn} .
Definition 1 component is a 2-tuple C = ( D, M ) , where: D = ( Dw, Dr , Dh) is a data set. Dw is the set of writable member variables to the external component user, Dr is the set of readable member variables, and Dh is the set of variables which are hidden within the component and can’t be accessed by external user. Here, the writable and readable member variables
1417
changed component firstly, and then traverse the whole MCG conversely from the changed method to calculate out all possible affected published methods. The process of computing affected methods can be described by the algorithm calculateCI in Figure 2. The time complexity and space complexity of this algorithm are O(| E |) and O(| V | + | E |) respectively, where | V | is the number of nodes in MCG (C ) and | E | is the number of edges.
3.2 Sample Program In this paper, we also use the example of VendingMachine in Reference [4] to explain our testing strategy, and the program code is omitted for space reason. That simple component-based system consists of a component (Dispenser) and an application (VendingMachine) that use it. (For further details concerning this system, that reference can be consulted.) Since Dispenser is a component, we assume its source code is unavailable to the system developer, conversely, the source code of the application VendingMachine is invisible to component developer. Take the component Dispenser for example, supposing that two changes will occur in this component respectively. (1) CP1: If the statement in line 54 of method dispense has been modified and we want this change point to be executed, the input parameters of this method must satisfy the following relation just before it is credit!=0 ∧ credit
