A Multipurpose Code Coverage Tool for Java - Semantic Scholar

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

A Multipurpose Code Coverage Tool for Java Raghu Lingampally, Atul Gupta, and Pankaj Jalote Indian Institute of Technology Kanpur, India [email protected], {atulkg,jalote}@cse.iitk.ac.in Abstract Most test coverage analyzers help in evaluating the effectiveness of testing by providing data on statement and branch coverage achieved during testing. If made available, the coverage information can be very useful for many other related activities, like, regression testing, test case prioritization, test-suite augmentation, test-suite minimization, etc. In this paper, we present a Java-based tool JavaCodeCoverage for test coverage reporting. It supports testing and related activities by recording the test coverage for various code-elements and updating the coverage information when the code being tested is modified. The tool maintains the test coverage information for a set of test cases on individual as well as test suite basis and provides effective visualization for the same. Open source database support of the tool makes it very useful for software testing research.

1

Introduction

Software testing is an essential activity for program validation. Given that the exhaustive testing of a program generally not possible, next best thing to do is adequate testing which is typically performed by obtaining a test model of the code under test (CUT), may be at different levels of abstraction, and covering some specific aspect of that model. Control flow graph (CFG) based coverage testing for structural entities (e.g. statements in the code) are among the most popular testing techniques used in practice. Bug fixes, code re-factoring, software-upgrades are most frequent activities in software development because of which code keeps on changing. The purpose of regression testing is to ensure that these changes in the code do not adversely affect the correct functionality inherited from its previous version. Agrawal et al. [16] demonstrated an effective approach for incremental regression testing of C programs by not selecting those tests which do not cover any of the changed statements of the code using a tool ATAC [26]. Vokolos and Frankl [30] applied textual differenc-

ing technique for regression testing on C programs and reported huge savings in time and size of the resulting regression test suites. Test-suite augmentation problem is to determine which new test cases must be developed to test added/uncovered functionality of the modified code [22]. For more on regression testing and test suite augmentation, please refer to [32, 21, 22]. Test case prioritization is an approach to ordering the test cases in terms of some effectiveness criteria which can be used further for other activities like enhancing regression testing [19, 28, 24], test suite minimization, etc. There are various approaches for test case prioritization, many of which make use of coverage information. For example, method/block coverage in [28]and branch/MCDC coverage in [24]. Efficient prioritization-algorithms are designed which make use the information of uncovered blocks at binary code level [28]. A study done by Wong et al. [32] suggested that test suite minimization may reduce test suite size remarkably, at little cost to the fault-detection effectiveness of those test suites, but later, Rothermal et al.[27] in another empirical study, found that the fault detection capabilities of test suites can be severely compromised by minimization. Such contradictory results from experimental research are not uncommon and suggest further explorations of the facts to increase our understanding of these approaches, for which availability of appropriate tools and other infrastructural support is absolutely essential[18]. The work described in this paper can be considered as one such effort in that direction. The main contributions of this paper are: 1. The tool that we introduce here is a Java bytecode coverage analyzer which provides test coverage information at method, block, branch and predicate level. 2. The tool supports change management by identifying changed (added, deleted or modified) code-elements, and correspondingly updating test code coverage for the changed code. 3. An important aspect of the tool is that it perform all the analysis at individual test case level, thereby facilitat-

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07) 0-7695-2755-8/07 $20.00 © 2007 1530-1605/07 $20.00 © 2007 IEEE

1


ing the detailed analysis and support for activities like test suite minimization, test case prioritization, regression testing, experimental validations, etc. 4. We demonstrate the usefulness of our tool by using two simple examples. 5. The tool has simple open source database interface for MySQL. The database can be access directly or through the tool to facilitate further exploration of the facts. We illustrate this point by discussing some of the studies done using the tool in this regard.

and therefore more effective than the other two. It can provide integral coverage for single statement at source code level (a source code line may contain multiple code statements) and can be easily integrated to run on the fly.

2.2

Existing Coverage Tools for Java

The paper is organized as follows: First we discuss coverage based testing and present a brief survey on some of the existing Java coverage tools in the Section 2. The tool JavaCodeCoverage organization and underlying approach is presented in Section 3, and its usefulness is demonstrated using two simple examples in Section 4. In Section 5, we highlight further prospective usage of the tool. We present the summary and future work in Section 6.

Table 1 presents a comparison between existing code coverage tools for Java including the one we introduced here. Most of the tools (Table 1) provide statement, branch, and/or method level coverage information and do not have support for change management. Many tools do not provide detailed coverage information and present a coverage summary only. Our tool provides detailed coverage information for all these program elements including predicates. The open-source database support with detailed test coverage information including coverage updates on code changes makes it a very useful tool for software testing research.

2

3

Coverage Based Testing

Code coverage analysis is the process of finding areas of a program not exercised by a set of test cases, thereby resulting a quantitative measure of code coverage, which is an indirect measure of code quality [23]. At the same time it can help in creating additional test cases to increase the coverage and may identify unreachable potions of the code. Additionally, it can identify redundant test cases that do not increase coverage, as well as help in testing changes made to the code during regression testing. A proper coverage tool is highly desirable to meet all these objectives.

2.1

Java Coverage Analyzers

Coverage measures based on various code-elements such as methods, statements, blocks, branches, predicates are most widely employed for coverage based testing. The coverage analysis tools are language dependent. Coverage analyzers work by adding probe instructions in the program which increment counters. These instructions are collectively called as instrumentation. Java coverage analyzers can be categorized according to the instrumentation techniques used, as: • Source code instrumentation • Bytecode instrumentation • Those that run the code in a modified JVM. Bytecode instrumentation adds probe instructions directly to the bytecode. It is very useful in situations where the source code is not available. It neither requires a modified JVM nor requires recompiling all the source code twice

JavaCodeCoverage Tool

JavaCodeCoverage is a bytecode analyzer tool for test coverage analysis for Java software which neither requires the language grammar nor the source code. An important aspect of JavaCodeCoverage is that it stores the coverage information for individual test case thereby facilitating detailed coverage analysis. Another important aspect of JavaCodeCoverage is that it records all vital code-elements and test coverage information in a open source database software MySQL [11]. This allows easy accessibility to stored information which increases the utility of the tool for many other purposes like those used in [20, 29]. Besides storing the information in the database, it also provides a friendly graphical user interface (GUI) to visualize typical code and test coverage information. Some other features of the tool are: • It records information regarding source code structure of the software including classes and their methods. • For each method, it records information about number of blocks, branches, predicates (conditions) including their location in source as well as in bytecode. • It records the changes made in the code compared to its previous version and accordingly update the test coverage information of the test suite executed on its previous version. This helps in performing regression testing effectively. • It also helps in identifying heavily tested coverage elements as well as lightly or un-tested coverage elements by recording their frequency of execution in the test case/test suite.

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07) 0-7695-2755-8/07 $20.00 © 2007

2


Coverage Tool

Type

Types of Coverage

Instrumentation type

GUI support

Hansel [1]

Open Source Open Source Proprietary

Statement

ClassLoader

Block, branch Statement, branch, method Line, block, method, class

Quilt [2] Clover [3] Emma [4]

Open Source

JavaCodeCoverage Open [5] Source JCover [6]

Proprietary

Jcoverage [7]

both options Proprietary

TCAT [8] Java Test Coverage [9] InsectJ [10]

Proprietary Open Source

Method, block, branch, predicate Statement, branch, method Line Branch, method-pair Branch Statement, branch, method call coverage

N

Coverage Detailing (Test case/ Test Suite) Test suite

Support for Change Management N

Direct Database Access N

Requirements

ClassLoader

N

Test suite

N

N

Source code

Y

Test suite

N

N

Jakarta packages, Antlr

instruments Bytecode off line or ClassLoader Bytecode

Y

Test suite

N

N

Y

Test case/Test Suite

Y

Y

Bytecode

Y

Test suite

N

Y

No external library dependencies BCEL 5.1, MySQL, GNU diff utility Java Debugger Interface

Bytecode

N

Test suite

N

N

NA

Bytecode

Y

Test suite

Y

N

NA

Source code

Y

N

N

NA

Bytecode

Y

Test case/Test suite Test case/Test suite

Yes

N

BCEL, JDK

BCEL JDK1.4 JDK 1.4

5.0,

Table 1. Coverage Tool Comparison • The tool can be augmented with Java testing frameworks like JUnit [12]. • It can be used as a command line or, alternatively, via its GUI browser. In this section, we describe the overall organization of the tool JavaCodeCoverage and underline its instrumentation and change analysis approaches.

3.1

JavaCodeCoverage Architecture

Functionally, the overall system has four components as shown in Figure 1. • • • •

Instrumentor Database Code Change Analyzer GUI

Code under test (CUT) is given to the instrumentation engine, i.e., the Instrumentor, which analyzes the bytecode using BCEL APIs [13] and inserts probes at appropriate locations depending on the type of coverage desired. The instrumented code is then run with a test case or a test suite. During a test-execution, when a probe is invoked, the relevant data is recorded in the database. When the program is changed, the changes are tracked during change analysis and decoded. The Code Change Analyzer is responsible for updating the coverage information

Figure 1. JavaCodeCoverage Components so as to effectively identify the portions of the changed program to be covered by further testing. This may be done by selecting relevant test cases from the original test suite and/or applying new test cases.

3.2

The Instrumentor

The bytecode of a class typically represents a unit of code for the Java-software. We analyze the whole software by doing the class-level bytecode analysis. For each class,


3


its bytecode is instrumented in two passes as shown in Figure 2 In the first pass, CFG for each method of the class is generated and appropriate probe locations for all types of coverage are identified. In the second pass, the class is instrumented using probes calculated in the first pass. The class is instrumented for four types of coverage, namely, method, block, branch, and predicate coverage. During instrumentation phase, probes are inserted which call static methods of execution monitors to store the coverage information in the database.

in the programs is handled separately. Branch probe consists of class-id, method-id, and branch-id. • Predicate coverage: In the first pass, all the predicates are identified. In second pass, a probe instruction is inserted to track the operands for predicate. A probe instruction is inserted before every simple predicate and branches are set appropriately. A predicate probe consists of information about predicateid, method-id, class-id, and instructions to duplicate predicate-operands on the stack. The process for code instrumentation and obtaining coverage information is outlined in Figure 2.

3.3

DataBase Organization

A MySQL database [11] is used to record the codeelements and test coverage information. Database organization can be categorized into three kinds of tables. (See Table 2)

Figure 2. Code Instrumentation for Coverage Execution monitors consist of static methods which update coverage information while instrumented program is running. Whenever a call to static method of execution monitor is made, the information about the call is buffered and call is returned immediately. Subsequently, the buffered information is transferred to database using database access layer. Database access layer is a set of classes which stores and retrieves the information from database persistently. The way in which probes are inserted into the program depends on the type of coverage. Here is a brief description of the places at which probes are inserted for each coverage type. • Method coverage: At the beginning of each method, a probe is inserted to track coverage information for method coverage. Here the probe consists of class-id and method-id. • Block coverage: A probe is inserted just before the first instruction of every block to track coverage information. This probe consists of class-id, method-id, and block-id. • Branch coverage: Probes are inserted just before a branch instruction and before the two branch targets to track branch coverage. Branch coverage of exceptions

1. Information Tables: These tables capture various code-elements information such as methods, blocks, branches, and predicates present in the program. They also maintain the code-change information about the instrumented classes from their previous versions, including source-line number, offset in the bytecode and indices which are used by the other tables. 2. Coverage Tables: These tables record over-all coverage information for different types of coverage which is updated dynamically with test-executions. 3. Test case Tables: These tables consist of test case information and execution details. Test case definitions consists of date of test case creation, tester, command used, input details and comments provided by the tester. Each execution or run of the program is considered as a test case. Coverage for each test case and overall coverage can be obtained separately. Coverage information and execution trace for each test case is also stored in these tables.

3.4

Code Change Analyzer

A change in the code, typically represented by code additions, deletions, and/or modifications in the code, results in a new version of that code. In this section we describe how code changes are analyzed and reflected in the database. We assume that the whole code is organized into a set packages arranged in a tree-structured hierarchy. We describe the approach for identifying changes at a package scope, as the complete change identification then would be analyzing all such packages which constitute the whole software. The Code Change Analyzer consists of a Change Decoder, a


4


S. No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Information Tables

Coverage Tables

Test Execution Tables

MySQL Tables

Description

PACKAGES CLASSES METHODS BLOCKS BRANCHS PREDICATES METHOD COVER BLOCK COVER BRANCH COVER TRUEPREDICATE COVER FALSEPREDICATE COVER MINUSPREDICATE COVER METHOD EXECUTION BLOCK EXECUTION BRANCH EXECUTION PREDICATE EXECUTION

package information of the CUT classes information of the CUT method information the CUT (method id,class id, etc) block information of the CUT (block id, method id,class id, etc) branch information of the CUT (branch id, method id,class id, etc) simple predicate information of the CUT (predicate id,method id,class id, etc) method coverage information and # of times a block is executed by test cases block coverage information and # of times a block is executed by test cases branch coverage information and # of times a block is executed by test cases predicate coverage information if it has been exercised as TRUE or 1(long values) predicate coverage information if it has been exercised as FALSE or 0 (long values) predicate coverage information if it has been exercised as -1(long values) sequential method execution information for a set of test cases sequential block execution information for a set of test cases sequential branch execution information for a set of test cases sequential predicate execution information for a set of test cases with their result

Table 2. Important Database Tables and their Descriptions for JavaCodeCoverage Change Analyzer and an Impact Analyzer as shown in Figure 1. Change Decoder finds the changes made to the code at class level. The effects of these changes are consolidated by the Change Analyzer by propagating them to include other sections of the code. The impact of these changes on test coverage data is analyzed through the Impact Analyzer. 3.4.1

Change Decoder

Change decoder finds the changes made to the code. We follow a combination of top-down and bottom-up approach for change identification in the two version of the code. Initially, we begin to look for the set of the classes in a package that has been changed (i.e. added, deleted, or modified) in the newer version and then analyze the methods of a class for the change identification in the similar way. Most of the changes are worked out at byte-code level except the case with the changes in a method-body, which uses a mixed approach at source and byte-code levels. Overall, the steps in identifying the changes in the two versions of the code are: • A class in the newer version, if not existed in the older version, marked1 as ‘added’. For a given class which existed in older version and not in the newer version, we marked it as ‘deleted’. • If a class existed in old and new version then we mapped the methods in the older version and new version by considering their signature so as to identify ’added’ or ’deleted’ methods. For other methods (i.e. common to both versions) of the class, we compared byte-codes of each method in the two versions. If they are found to be different at any stage, the method is marked as ‘modified’. • For a modified method, we found the source line numbers (in the new version) which are added, 1 please note that the word ‘mark’ stands for updating the status for the relevant code-element in the database

deleted, and/or modified from the source code using GNU diff utility [14]. We mapped this information to the byte-code block positions of the older version of the method to determine the information of added/deleted/modified/unchanged blocks in the newer version. Please note that the default marking for the codeelements in the database prior to performing change analysis was ’unchanged’. So if an element is not marked during change analysis then it remained as ‘unchanged’. For added and modified blocks in the newer version, we compare branches and predicates. As every block ends with a branch condition, so any branch pointing to added/deleted/modified block is also marked as ‘modified’. For deleted blocks in the older version, we mark predicates and branches in that block as ‘deleted’. 3.4.2

Change Analyzer

Changes made to the CUT are further analyzed and categorized as shown in Table 3. Whenever a class is changed, the class and the package containing it are considered as changed. Whenever a method is changed, the method, the corresponding class, and the package containing that class are also considered as changed. Whenever any block/branch/predicate is changed the corresponding method, class and package are considered as changed. Bytecode of two versions of the classes are parsed and compared to find the differences at class, method, block, branch, and predicate level. For example, method level differences are computed by comparing all the methods of the new version of the CUT with the methods available in old version. If part of the code of a method in the new version is not matched with old version then it is considered as modified. For all such modified methods (i.e. with added, deleted, and/or modified code), code differencing at block, branch and predicate level is computed. If a method in new


5


version is not found in old version, it is considered as an added method. If a method in old version is missing in the new version, it is marked as deleted. Likewise, bytecode differences are computed for other levels. This changed information is used by the impact analyzer for resetting the test code coverage information. 3.4.3

Impact Analyzer

It analyzes the impact of changes made to the code and resets the test coverage appropriately. It also marks the execution details for test cases as un-covered for those codeelements which get affected by these changes. Impact on Coverage : After a change in the code, the test coverage for the new version is accordingly updated. The coverage of the changed blocks is reset, i.e., initialized to zero. The coverage information for unchanged part of the code is retained. For simplicity, we reset the coverage for the changed blocks only though more involved coverage re-setting is also possible in which control flow for the method is considered and coverage information for the elements which are successors of changed elements in the control flow will be reset. Method coverage for the method is reset when part of the code of a method is changed. When a method is changed the impact analyzer analyzes changes made to the method at block, branch, and predicates level. The approach for identification of change is a more conservative one in which source line numbers of the old version of code are identified for change management. The bytecode blocks of the old version corresponding to changed source line numbers are marked as modified, if they are also present in the new version of the code. Coverage is retained for the blocks which are unmodified and the coverage for the modified and added blocks is reset. Branch coverage for the branches which lead to the changed blocks (i.e. added, modified, or deleted) is reset and these branches are marked as modified. Coverage for all the predicates which are mapped on changed source line number is reset to zero and they are marked as modified. Accordingly, the coverage is updated for the reorganized block structures of the modified methods in the new version of the code. Impact on test cases : Test cases which covered the modified elements of the code in the older version, are the desired candidates for this analysis. Test case coverage information for the new version of the CUT is computed and reflected in database. This is done by re-setting the coverage for the affected code elements as obtained in impact analysis. This forms the basis for regression testing and test case prioritization as the desired information can be obtained simply by querying the database. A similar approach for identifying the changes in Java programs, Herrold et. al. [22] have used a CFG for mod-

eling a class method. They identified the ‘dangerous’ entities as those branches of the CFG branch targets which contained any changed statements and behave differently from the previous version of the code. Effectively all kinds of code changes are mapped on a set of affected branches in some method. Then they generate a regression test suite by selecting the test cases, from the test suite of the previous version of the code, which exercised these dangerous edges. Here we provide more flexibility by recording not only the affected branches but blocks and predicates as well, which may help in performing more sophisticated change analysis. Moreover, our tool also supports for test suite augmentation problem [22] by providing the information of newly added code elements. Herrold’s approach considered a method call as a branch in the CFG and therefore was able to perform interprocedural change analysis. As of now inter-procedural change analysis is not incorporated in our tool. This we consider as one of the future work to enhance the capability of our tool.

Figure 3. Change and impact analysis

3.5

GUI

The tool has a friendly GUI implemented using JAVA Swing. It retrieves code-elements, test coverage, and testexecution information from the database. The information for all the coverage measures is displayed at package, class, and test case levels. Updated test coverage information can also be displayed for the changed versions of the CUT. A snapshot of the tool’s GUI is shown in Figure 4.


6


Change Added

Package A new package is added

Class A new class is added

Deleted

A package is deleted

A Class is deleted

Modified

A class in package is modified or some classes added or deleted or modified

A class is modified or base class for the class or inheritance changed

Method New method is added to class Existing method is deleted Part of the method is changed

Block A new block is added to the control flow of method An existing block in the method is deleted

Branch New branch condition is added in the control flow

Instructions in the block modified added or deleted

Conditions relating to the branch or branch target changed

Existing branch condition is deleted

Predicate New predicate is added Existing predicate is deleted Predicate or its operands changed

Table 3. Code Change Categorization

Test cases and Instrumented classes

Classes executed by the test case #6

Block coverage highlighted for test case #6

Coverage summary for the test case #6 for class GameMaster

Figure 4. A snapshot of the GUI

4

JavaCodeCoverage: Illustrative Examples

In this section, we demonstrate the usefulness of our tool using two simple examples. Some other prospective usage of the tool is given in Section 5. Please note that the use of JavaCodeCoverage is not limited to the applications discussed in this paper.

4.1

Coverage Based Testing: An Illustrative Example for bug identification

In this experiment, we show how effective are the three coverage measures, namely, block, branch and predicate coverage, at detecting faults during testing. For this, we have taken simple hotel management program, HotelBooking, having six classes. Twenty bugs were inserted in the program according to a set of applicable mutation operators [25]. Using JavaCodeCoverage with 90% coverage for the respected measures as stopping criteria, the program was tested by three testers having similar experi-

S. No 1 2 3 4 5 6 7 8 9

Mutation Operator (Bug Type)

Abbr.

# Bugs

Unreachable code Initialization operator Literal Change Operator Language Operator Replacement Argument Change Order Method Name Replacement Variable Replacement Operator Statement Swap Operator Control Flow Disruption

URC ITO LCO LOR ACO MNR VRO SSO CFD

1 2 3 3 1 1 3 1 3

Table 4. Bug Statistics for HotelBooking

ence. Table 4 shows the information about the bugs seeded in the program. Bugs revealed by the random test suites for each coverage type is given in Table 5. The results suggest that predicate coverage was the best in terms of number of faults identified but worst in the efforts requirements (taken 23 test cases) whereas block coverage performed the other way round. Branch coverage appeared as a better compromise in this case.


7


Coverage Type Total-Bugs Block Coverage Branch Coverage Predicate Coverage

URC 1 0 1 1

ITO 2 0 1 1

LCO 3 2 3 3

LOR 3 2 2 3

Bug Type ACO MNR 1 1 0 1 0 1 0 1

VRO 3 3 3 3

SSO 1 1 1 1

CFD 3 2 2 3

Total 18 11 14 16

Test Suite Size 13 16 23

Table 5. Bug detected by Coverage type Package Number of Classes Number of Methods Number of Blocks Number of Branches Number of simple predicates Time taken for instrumentation

Jakarta RegExp 11 179 727 699 277 5582 ms

Table 6. RegExp Package Details Coverage type

No. units

Class Method Block Branch Predicate

11 79 727 699 277

of

No. of Covered units 11 53 451 420 165

Coverage % 100 53 67 60 40

Table 7. Coverage for RegExp V 1.0

4.2

Testing Changes Made to the Code: Regression Testing

In this experiment four versions of Jakarta’s RegExp package [15] consisting of 143 test cases for versions 1.0, 1.1 and 1.2 and 215 test cases for version 1.3 are analyzed for version changes at block coverage. After code change, some test cases become invalid and some test cases need to be re-executed. Regression testing was done so as to execute maximum portion of changed code and achieve maximum coverage. Table 6 shows the RegExp package details and time taken to instrument RegExp version 1.0. Table 7 - 10 show code coverage updates after each version change. For every next version, change impact on coverage is computed and recorded. Then the test cases from the older version are prioritized according to maximum block change coverage and executed on the newer version. Results show that coverage stabilizes after executing around 30 percent of prioritized test cases.

5

More on JavaCodeCoverage

Besides providing coverage information for various code-elements, or supporting change management, the tool can also be used for many other useful purpose like strengthen static analysis, computing code coverage information for other coverage criteria, efficient de-bugging,

program comprehension, mutation testing, etc. We present the case for such analysis in the following sub-sections.

5.1

Test Suite Minimization

From the test case coverage information for various code elements, we can reduce the size of the test suite by identifying and removing redundant test cases from it. Test suite minimization can be effectively used for comparing various coverage criteria. Gupta and Jalote [20] have compared the fault detecting effectiveness and efforts requirements of the three control coverage criteria, namely, block, branch and predicate coverage by randomly generating coverage adequate minimal test suites for these criteria. They also reported that block test suites’ performance index was the highest but with greatest variability whereas performance index for predicate test suites was lowest with least variation. This diagnosis was possible as they were able to compute minimal test suites using the test coverage information supplied by JavaCodeCoverage.

5.2

Strengthen Static Analysis

Static analysis is a complementary approach to dynamic testing for bug identification. The code is scanned and discrepancies like null pointer referencing are identified. Exhaustive static analysis of the program through all paths is not always possible, especially for a large software causing incompleteness in the analysis. The complexity of the static analysis can be reduced by effective program partitioning. Vipindeep and Jalote [29] described a technique which uses coverage data from testing to remove the tested paths and then statically analyzes the remaining code. This pruning of tested paths allowed a static analyzer to perform a more predictive and thorough analysis of the reduced code, which led to considerable improvements in its effectiveness.

5.3

Extending Coverage Measures

From existing coverage information, some other related coverage measures like Condition/Decision, Relationaloperator, and MCDC, can be measured without reinstrumenting or parsing the class file. The desired information can simply be obtained by making suitable queries to


8


Coverage Type Class Method Block Branch Predicate

No.of units 14 108 801 764 298

Modified units 11 3 130 143 68

Added units 3 29 74 65 44

Coverage after version change 79 49 (53/108) 40 (321/801) 36 (277/764) 25 (74/298)

Coverage after prioritized test case execution 100 55 (60/108) 58 (467/801) 55 (426/764) 39 (116/298)

Table 8. Coverage after version change RegExp V 1.0 to RegExp V 1.1 Type Class Method Block Branch Predicate

No.of units 14 108 806 764 298



Coverage after version change 100 49 (52/108) 49 (396/806) 45 (349/765) 25 (74/298)


Table 9. Coverage after version change RegExp V 1.1 to RegExp V 1.2 the database. Here is a brief description of how to compute the new coverage measures with the existing framework. Condition/Decision coverage : Condition/Decision coverage requires every simple condition (predicate) to be evaluated both as true and false as well as both the outcomes for the conditional expression are to be exercised. This information can easily be obtained from the coverage tables in the database. Relational operator coverage : An information table for predicates contains opcode for every simple predicate. To measure relational-operator coverage, only simple predicates with relational-operator opcodes are to be considered in the database queries. MC/DC coverage: The Modified Condition Decision Coverage (MC/DC) reports coverage of occurrences in which triggering individual simple condition should also trigger (i.e. flip) the overall decision. The above information can be inferred from execution and information tables for predicates.

5.4

Efficient De-bugging

The tool can effectively used for bug-place identification during testing. The GUI of the tool displays the code coverage for each test case exercised manually or in batch mode. From the test code coverage information, one can quickly get to the potential buggy places.

5.5

Program Comprehension

The test cases execution traces can be used for generating the behavior models like state diagrams for objects of the CUT. This helps in program comprehension and testing activities. Recently, Dallmeier et al. [17] used execution traces from a set of test cases and constructed objects’ state models with pre-defined abstract states. Similar work is also reported in [31].

6

Summary and Future Work

Bug fixes, code re-factoring, software-upgrades are most frequent activities in software development because of which code keeps on changing. Due to frequent code changes, and also in a resource constraints environment, one needs to optimize the overall testing efforts which typically include effective techniques for regression testing, test case prioritization, test case selection, test suite augmentation, and test suite minimization. These techniques mostly rely on code coverage information for various codeelements. In this work, we presented a Java based coverage tool JavaCodeCoverage, which records class, method, block, branch and predicate coverage measures for the tests applied on a CUT. The tool performs the analysis at bytecode level which is more effective than other source code instrumentation tools. Additionally, the tool performs the code change analysis at the bytecode augmented with source code change information and updates the test code coverage information for the newer version of the code. This information can be very useful for performing other testingrelated activities. The tool uses a MySQL database to store code elements and test coverage information at the granularity of individual test case. Using a set of experiments, we have demonstrated the use of JavaCodeCoverage for Coverage testing, Code change tracking and test case prioritization. Several other studies done using JavaCodeCoverage clearly highlight the utility of our tool in other fields of software testing. Open source database support of the tool makes it a very useful for software testing research. We further plan to strengthen the capabilities of JavaCodeCoverage by incorporating other coverage measures like MC/DC, simple path coverage and data flow coverage. We are also considering more sophisticated code change analysis such as control flow based inter-procedural analysis for coverage updates. The tool can be downloaded


9


Type Class Method Block Branch Predicate

No.of units 14 110 754 862 346



Coverage after version change 100 34(37/110) 21 (161/754) 21(184/862) 12 (41/346)


Coverage after additional test cases execution 100 58 (64/110) 64 (487/754) 42 (364/862) 40 (139/346)

Table 10. Coverage after version change RegExp V 1.2 to RegExp V 1.3 from [5] with other information.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

[17]

[18]

[19] [20]

[21]

Hansel. http://hansel.sourceforge.net/. Quilt. http://quilt.sourceforge.net/. Clover. http://www.cenqua.com/clover/. EMMA. http://emma.sourceforge.net/. JavaCodeCoverage. http://www.cse.iitk.ac.in/ users/atulkg/. JCover. http://www.codework.com/JCover/product. htm. Jcoverage. http://jcoverage.com/products/ jcoverage-commercial.html. TCAT for Java. http://www.soft.com/TestWorks/. Java Test Coverage. http://www.semdesigns.com/ Products/TestCoverage/JavaTestCoverage.html. InsectJ. http://insectj.sourceforge.net/. MySQL. http://www.mysql.com/. JUnit Home Page. http://www.junit.org . BCEL. http://jakarta.apache.org/bcel/. GNU diff utility. http://www.gnu.org/software/ diffutils/. Jakarta RegExp Package. http://jakarta.apache.org/ regexp/. H. Agrawal, J. R. Horgan, E. W. Krauser, and S. London. Incremental regression testing. In ICSM ’93: Proceedings of the International Conference on Software Maintenance, pages 348–357, Washington, DC, USA, 1993. IEEE Computer Society. V. Dallmeier, C. Lindig, A. Wasylkowski, and A. Zeller. Mining object behavior with adabu. In WODA ’06: Proceedings of the 4th Workshop on Dynamic Analysis, pages 17–24, New York, NY, USA, 2006. ACM Press. H. Do, S. Elbaum, and G. Rothermel. Infrastructure support for controlled experimentation with software testing and regression testing techniques. In ISESE ’04: Proceedings of the 2004 International Symposium on Empirical Software Engineering, pages 60–70, Washington, DC, USA, 2004. IEEE Computer Society. S. Elbaum, A. G. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Eng., 28(2):159–182, 2002. A. Gupta and P. Jalote. An experimental comparison of control flow based coverage criteria on seeded faults. In TACAS ’06: Proceedings of the 12th. International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 365–368, Heidelberg, Germany, 2006. Springer. R. Gupta, M. J. Harrold, and M. L. Soffa. Program slicingbased regression testing techniques. Software Testing, Verification, and Reliability, 6(2):83–111, 1996.

[22] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for java software. In OOPSLA ’01: Proceedings of the 16th ACM SIGPLAN conference on Object Oriented Programming, Systems, Languages, and Applications, pages 312–326, New York, NY, USA, 2001. ACM Press. [23] P. Jalote. An Integrated Approach to Software Engineering. Addison-Wesley, 2002. [24] J. A. Jones and M. J. Harrold. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng., 29(3):195–209, 2003. [25] S. Kim, J. Clark, and J. McDermid. The rigorous generation of java mutation using hazop. In ICSSEA ’99 : Proceedings of the 12th International Conference on Software & Systems Engineering and their Applications, pages 9–10, 1999. [26] M. Lyu, J. Horgan, and S. London. A coverage analysis tool for the effectiveness of software testing. IEEE Trans. on Reliability, 43(4):527–535, 1994. [27] G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In ICSM ’98: Proceedings of the International Conference on Software Maintenance, pages 34–43, Washington, DC, USA, 1998. IEEE Computer Society. [28] A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In ISSTA ’02: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 97–106, New York, NY, USA, 2002. ACM Press. [29] V. Vipindeep and P. Jalote. Efficient static analysis with path pruning using coverage data. In WODA ’05: Proceedings of the 3rd Workshop on Dynamic analysis, pages 1–6, New York, NY, USA, 2005. ACM Press. [30] F. I. Vokolos and P. G. Frankl. Empirical evaluation of the textual differencing regression testing technique. In ICSM ’98: Proceedings of the International Conference on Software Maintenance, pages 44–53, Washington, DC, USA, 1998. IEEE Computer Society. [31] J. Whaley, M. C. Martin, and M. S. Lam. Automatic extraction of object-oriented component interfaces. In ISSTA ’02: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 218–228, New York, NY, USA, 2002. ACM Press. [32] W. E. Wong, J. R. Horgan, S. London, and A. P. Mathur. Effect of test set minimization on fault detection effectiveness. In ICSE ’95: Proceedings of the 17th International Conference on Software Engineering, pages 41–50, New York, NY, USA, 1995. ACM Press.


10

A Multipurpose Code Coverage Tool for Java - Semantic Scholar

A Multipurpose Code Coverage Tool for Java - Semantic Scholar

Suggest Documents

Dynamic Code Evolution for Java - Semantic Scholar

Dynamic Code Evolution for Java - Semantic Scholar

JaBUTi: A Coverage Analysis Tool for Java ... - CCSL/ICMC - USP

CC4J â Code Coverage for Java A Load-Time ... - Semantic Scholar

A Java Coordination Tool for Web-service ... - Semantic Scholar

A Visual Tool for Teaching Java Technology - Semantic Scholar

VisualJVM: A Visual Tool for Teaching Java ... - Semantic Scholar

JASMINE: A Java Tool for Multimedia ... - Semantic Scholar

a Tool for Detecting Java Card Illegal Flow - Semantic Scholar

A Visual Tool for Teaching Java Technology - Semantic Scholar

The PACAP Prototype: a Tool for Detecting Java ... - Semantic Scholar

Minik: A Tool for Maintaining Proper Java Code Structure*

Frances: A Tool For Understanding Code Generation - Semantic Scholar

M254 MULTiPURPOSE TOOL CaRRiER

Monitoring Java Code Using ConGu - Semantic Scholar

MPJ Express Meets Gadget: Towards a Java Code ... - Semantic Scholar

Plagiarism Detection Engine For Java Source Code - Semantic Scholar

Java-MaC: a Run-time Assurance Tool for Java ... - Semantic Scholar

Java PathExplorer - A Runtime Veri cation Tool ... - Semantic Scholar

A Tool for Semantic Test Coverage - Formal Methods and Tools

M254 MULTiPURPOSE TOOL CaRRiER - Shindaiwa

ASM: a code manipulation tool to implement ... - Semantic Scholar

A New Multipurpose Comprehensive Database for ... - Semantic Scholar

RNA-CODE: A Noncoding RNA Classification Tool ... - Semantic Scholar

A Multipurpose Code Coverage Tool for Java - Semantic Scholar