Branch coverage testing using anti-random technique (PDF Download ...

i-manager’s Journal on Software Engineering Volume: 8 No. 2

Issue : Oct-Dec 2013

Pages : 1-6

Branch Coverage Testing Using Anti-Random Technique Hashim J. Hasan * Mohammad A. Alshraideh ** Basel A. Mahafzah *** *-**-*** Department of Computer Science, King Abdullah II School for Information Technology, The University of Jordan, Amman, Jordan.

Abstract Software testing is one of the most important, costly and time consuming phase in software development. Anti-random testing chooses the test case where it's total distance from all previous test cases is the maximum, using the Hamming distance and Cartesian distance as measures of difference. In this paper, the authors present an anti-random technique to achieve high branch coverage in white-box testing, depending on the hypothesis that any two test values with small distance mostly discover the same errors and faults. Experimental results show that anti-random testing yields acceptable results, but the target of branch coverage is not achieved in all cases. We executed the algorithm 60 times over ten different programs, and they found that coverage achieved for eight programs runs with high performance in terms of execution time.

Keywords : 

Software Testing,



White Box Testing;



Branch Coverage;



Random Test Generation;



Anti-Random Test Generation.

Introduction Now-a-days computers and its software are spreading into most of our life, and change the way of all human activities. Computerizing and automating any work has become the solution of any hard, risky, costly and time consuming actions done by humans. It is the goal of founding such machines, that improve our life to be easier with more productivity. Due to building large scale and high complexity software, “Bugs” are always found and mostly may not affect the overall operation of the software, or they have just slight effects and can be controlled. But in other cases, even small and seemingly harmless bug can destroy equipment valued millions of dollars. Testing in software industry is a difficult and timeconsuming work; it consumes about 50% of software development cost (( Alshraideh, & Bottaci, 2006), ( Alshraideh, Mahafzah, & AlSharaeh, 2011), ( Alshraideh, Bottaci, & Mahafzah, 2010 ), ( Alshraideh et al.., 2013)). Automated software testing was introduced to significantly reduce the cost and the time of testing process. Hence, building and improving techniques and algorithms for automation software testing are subjects of research, where all algorithms target to achieve more software testing coverage, which lead to more errors discovered and resulting in high software confidence. So, in software testing we aim to reveal such errors in order to avoid failures during program usage. We consider program's execution is correct, when its behavior matches the functional and non-functional requirements specified by software users ( Alshraideh et al., 2011). In general, testing technique has three main categories: Black-box (functional testing), where test cases are developed and designed according to the system requirements and

functionality. White-box (structural testing) is based on the knowledge of the internal logic and design of the application. Gray-box testing and statement coverage combines white-box technique with black-box input testing(( Moataz, & Hermadi, 2008), ( Al- Ashhab, 2010 )). Generating test cases manually is hard and expensive. Nevertheless, for any system development lifecycle, this activity is very important and cannot be avoided ( Moataz, & Hermadi, 2008 ). Random methods are considered to generate test cases, but lack of reliability and enabling to achieve the required coverage of the software are the main direct aspect of them. The advantages are its low cost and the ability to generate several different test cases automatically in the absence of the source code details ( Al-Ashhab, 2010). It can detect defects and bugs which cannot be discovered by deterministic approaches. The disadvantage is it may produce large number of test cases not applicable or not executable by missing any functional logic. The test designer and creator have no control over the generation of the test cases; so they may not accept test coverage ( Alshraideh, Mahafzah, & Al-Sharaeh,2011). Therefore, we need to improve random method to be more accurate. So, in this paper the authors use antirandom test generation. The assumption is that similar input to a system will expose similar types of bugs; therefore it is useful for an input set to be as different as much as possible. Each test applied is chosen such that its total distance from all previous tests is the maximum ( Shen et al., 2008). Hamming distance and Cartesian distance are used for measure of difference. The input variable for a program converted to binary, allows binary anti-random sequence to be decoded into actual inputs. In such cases, anti-random testing is used to find defects sooner, and reducing the overall testing and debugging time ( Kulvinder, Rakesh, & Kaur, 2010). In such cases, we can maximize the effectiveness of testing by improving the performance of computation by computing first “distance” value and then using it as reference to construct whole anti-random sequences ( Yashwant, & Malaiya, 1997). In this paper, the authors have applied anti-random technique on branch coverage testing, which means that every condition in the program is tested with both accessible results, true and false. It is "a measure of the percentage of the decision points (Boolean expressions) of the program that have been evaluated as both true and false in test cases" ( Alshraideh, Mahafzah, & Al- Sharaeh,2011 ). Achieving coverage of any branch means; executing the program with test cases, which make the whole branch result as true and another execution that makes branch result as false.

The remainder of this paper is organized as follows: Section 1 presents basic concepts of Test Data generation. Automatic testing is presented in Section 2. In Section 3, Automatic test-data generation using antirandom technique to test named block in Oracle is presented. The experimental results have been presented and discussed in Section 4. Finally conclusion Section and some suggestions for future works are also given.

1. Test-Data Generation When we are talking about software testing as a research subject, generating good testing data is the main point in this field. All algorithms and techniques try to generate suitable test cases from finding errors and bugs perspective. A variety of methods have been developed to generate testing data or test cases over the years. Each method intends to satisfy certain test criterion to generate better test cases. At the end the more bugs discovered by test cases, the better test cases are. In general, the process of automatic structural test data generation, for branch coverage, consists of three major steps ( Yin, Lebne-Dengel , & Malaiya 1997): 

Build control logic graph, as CFG or CDG.



Determine the target according to testing coverage criteria.



Finding the set of test data that satisfies the selected criterion. The target program must be instrumented in order to control the evaluation of testing objective when the program under test is executed with given input data. In most test data generators, the program must be instrumented as pre-process stage before the generator can actually be used This instrumentation process is the process of inserting probing statements at the beginning of every block of code of interest, i.e. at the beginning of each branch predicate. For example, in branch coverage, these statements are used to provide a feedback to the generator about the reached branch within the program while it is executed with trial test data ( Alshraideh et al.., 2013). The approaches of generating testing data can be classified into three categories, Random test data generation, Goal oriented test data generation and path oriented test data generation.

1.1 Random test-data generation Random testing is the simplest method of generation techniques. It could actually be used to generate input values for any type of program since, ultimately, a data type such as integer, string, or double is just a stream of bits. Therefore, for any function that takes a string as an

input argument, we just generate a stream of bits randomly and consider it as input string ( Alshraideh et al., 2013 ). However, random testing mostly does not generate good testing data in terms of coverage. However, it depends on the probability of a quite low chances in finding small faults, thus accomplishing high coverage. Small fault is a fault that is only discovered by a small percentage of the program input ( Alshraideh et al.., 2013). Consider, for example, the piece of code in Figure 1.

Figure 1. Example of Random Test Data Generation

The probability of exercising the write (1) statement is 1/n, where n is the number of values in the input domain of integers, where executing this statement, needs variables a and b to be equal. We can easily imagine that generating even more complex structures than integers will give us even worse probability.

1.2 Goal-oriented test-data generation The goal-oriented approach performs better than random generation, in the sense that it provides guidance towards a certain set of paths. Instead of letting the generator generate input that traverses the path from the entry node to the exit node of a program, it generates input that traverses an unspecific path. Because of this, it is sufficient for the generator to find input for any path. This inturn reduces the risk of encountering relatively infeasible paths and provides a way to direct the search for input values as well ( McMinn, Binkley., Harman , & Tonella 2006).

1.3 Path-oriented test-data generation Path-oriented generation is the strongest among the three approaches. It does not provide the generator with a possibility of selecting among a set of paths, it rather considers one specific path. In this way, it is the same as a goal-oriented test-data generation, except for the use of specific paths. Successively, this leads to a better prediction of coverage; in other hand, it is harder to find a test-data. Since they are solely based on the CFG, they often lead to the selection of infeasible paths ( Alshraideh et al., 2013 ).

2. Automatic testing The process of generating test-data to achieve the required coverage of a given program manually is labor intensive, expensive and time consuming. Thus, development techniques that support the automation of software testing will be helpful and a cost saving process. When we are looking for the best solution among set of solutions, the space of all possible solutions is called search space. And each solution in the search space acts as a candidate solution. Searching for best solution in a pool (domain/set) of possible input data that satisfies test adequacy criteria is a search problem; for example force reaching a has its own advantages and disadvantages over the others. They are strongly problem specific branch. Therefore, much effort has been spent to automate software testing ( Alshraideh et al., 2013). Accordingly, meta-heuristic search algorithms proposed a potential better alternative for developing test data generators ( Alshraideh et al., 2013) Efficient existing meta-heuristic search algorithms include Simulated Annealing (SA), Taboo Search (TS), GA and Ant Colony Optimization (ACO).Each of these search algorithms are domain dependent, because they use domain dependent knowledge or heuristics related to domain of the problem under consideration.

2.1 Random test generation Random test generation is randomly selecting test cases/sequences of events from the input domain or test data domain. The advantages of random test include its low cost, ability to generate several different test cases automatically and generation of test cases in the absence of the software specification and source code details (Al-Ashhab, &Majd, 2010). These mentioned advantages lead to randomness for the testing process. Such randomness can be the best reflect of the real system operational environment; therefore random testing can detect defects and bugs which can not be discovered by deterministic approaches. All these advantages make random testing initial, basic, and central testing technique in any testing environment or plan ( Shen et al., 2008). On the other hand, this approach may produce large number of test cases that are not applicable or not executable by missing any functional logic. Moreover, the test designer and creator have no control over the genseration of the test cases; so they may not accept test coverage ( Kulvinder, Rakesh, &l Kaur, 2010). Random testing selects arbitrarily test data from the input domain, then these test data are applied to the intention program, so the chances of testing a function or a part of this

program is tightly related to the input data sub domain that is used by this function, thus the probability of reach of this part is not matured ( Shen et al., 2008). A random generation of test cases generates test data with no use of feedback from previous tests. The tests are passed to the procedure under test, in the hope that all branches will be traversed ( Shen et al., 2008).

2.2 Adaptive random test generation Adaptive random testing is an enhancement of random testing. It has been introduced to improve the effectiveness of the fault detection of random approach, for the situations where the reasons of the failures inputs are clustered together ( Kulvinder, Rakesh, &l Kaur, 2010). Such situations frequently occur in real life programs, when the reasons of the failures are concentrated in certain regions, known as the failure regions, and keeping test cases a side shall enhance the effectiveness of random testing. Therefore, adaptive random testing does not just randomly generate test cases, but also spreads them to generate fewer duplicate test cases. Different studies show that adaptive random testing can be very effective in detecting failures, when there is a continuous failure region inside the input domain as compared to random testing. Since adaptive random testing is as simple as random testing and preserves certain degree of randomness. Adaptive random testing could be an effective replacement of random testing ( McMinn, 2004).

2.3 Pseudo-random test generation In pseudo-random testing, each test vector in the test cases domain is chosen with equal probability out of a pool that initially contains M different vectors ( Shen et al., 2008 ). A common approach is to generate them using an Autonomous Linear Feedback Shift Register (ALFSR) based on a primitive polynomial ( Shen et al., 2008). Pseudo-random testing does not use some information that is available in black-box testing environment. This information consists of the previous tests applied. If an experienced test engineer is generating tests by hand, he would select each new test such that it covers some part of the functionality not yet covered by tests already generated ( Shen et al., 2008).

2.4 Anti-random test generation The concept of anti-random testing is in which each test applied is chosen such that its total distance from all previous tests is maximum ( Kulvinderh, Rani, & Rekha, 2011 ). This

information of executed test cases is available during black box testing, but random testing does not explore that information. Some challenges are facing anti-random testing, such as generating anti-random sequence related to Boolean inputs, where computing the distance for each new test is not applicable, as we use Hamming Distance and Cartesian Distance for measure of difference. In general, the input variable for a program can be numbers, characters ( Yashwant, & Malaiya, 1997). In anti-random testing we convert input code into binary, then this code allow binary antirandom sequence to be decoded into actual inputs. In such cases, anti-random testing finds defects sooner, and reduce the overall test and debugging time. Research studies have shown that pure random testing is relatively less effective at discovering bugs than other testing paradigms; however, random testing is likable because it is typically quick and easy to implement. The idea of anti-random testing has been introduced by the field of hardware testing. Essentially, anti-random testing generates an input set of random values, where the values are as different as possible from each other. The assumption is that similar input to a system will expose similar types of bugs; therefore it is useful for an input set to be as different as much as possible. Following is a list of terms used in anti-random technique: 

Anti-random test sequence: it is an Anti-random test sequence such that Ti is satisfied with some criteria with respect to all test to Ti…..Ti-1.



Distance: measurement of different two vectors like Ti and Tj.



Hamming distance: that is also the test sequence that is the numbers of bits in which two binary numbers differ.



Cartesian distance: that is the difference between two vectors. For example, consider vectors A and B as follows: A={an, an-1, an-2… a1, a0}and B={bn, bn-1, bn-2… b1, b0}, where an, an-1, an-2… a1, a0 and bn, bn-1, bn-2… b1, b0 are elements of A and B, respectively. Cartesian distance is given as:

As the two test vectors are binary, then Cartesian distance can be written as:



Total hamming distance: in any number is the sum of its hamming distance with respect to all previous numbers.



Total Cartesian distance: in any number is sum of its Cartesian distance with respect to all previous numbers.



Maximal Distance Anti-random Test Sequence (MDATS): it is a test sequence such that each test Ti is chosen to make the total distance between Ti and each of to, Tl ... Ti-1 is maximum for all possible choices of Ti. We used Hamming distance and Cartesian distance to construct Maxima Hamming Distance Anti-Random Sequence (MHDATS) and Maximal Cartesian Distance Anti-random Sequence (MCDATS). Figure 2 shows an example of construction of 3-bit MHDATS:

Figure 2. Construction of 3-bit MHDATS

In 1997 Malaiya Huifang proposed checkpoint encoding for anti-random in order to reduce the cost of generating anti-random sequence ( Al-Ashhab, & Majd, 2010). Checkpoint encoding is the process of representation of any input domain of a software system into a binary valued domain. Any input data value can now be translated with minimal loss of information into a binary valued string. Anti-random testing is a scheme that takes the advantage of binary representation of the data, where checkpoint encoding concept is used and shows good results. The checkpoint encoding process is usually applied manually and in an arbitrary fashion. The result varies depending on the choices made by individual test engineer carrying out the process ( AlAshhab, &Majd, 2010 ). In most of cases, inputs can be numbers, alphanumeric characters, and data structures, with ability to compose them. In such cases, we can maximize the effectiveness of testing by improving the performance of computation by computing first “distance” value and then using it as reference to construct whole anti-random sequences (Al- Ashhab, &Majd, 2010 ). Fast anti-random is presented as a technique to generate anti-random sequence in a faster computation method, where this method depends on computing anti-random vector that

differs as much as possible from all previous test cases, which is computationally feasible ( Chen , Andre , & Hajjar, 2002 ).

3. Automatic test-data generation using anti-random technique to test named block in Oracle Automatic test data generation is a process of proposing automatically a “good” set of test cases for a program to be tested. This process is very useful and helpful for the time consuming and heavy tasks of testing in software development cycle. The automatic generation of test cases for computer programs has been tackled as research area since many years ago, hence great number of paradigms have been applied to the test data generation.

3.1 PL/SQL named block There are two types of named blocks in PL/SQL, which are procedures and functions. Every named block will have a declarative part, an executable part or body, and an exception handling part, which is optional. Declarative part contains variable declarations. Body of a named block contains executable statements of SQL and PL/SQL. Statements to handle exceptions are written in exception part. Subprograms provide the following advantages: 

They allow you to write PL/SQL program that meets our need.



They allow you to break the program into manageable modules.



They provide reusability and maintainability for the code. Procedure is a subprogram used to perform a specific action. A procedure contains two parts: specification and body. Procedure specification begins with keyword CREATE and ends with procedure name or parameters list. Procedures that do not take parameters are written without a parenthesis. The body of the procedure starts after the keyword IS or AS and ends with keyword END. A function is a named PL/SQL Block which is similar to a procedure. The major difference between a procedure and a function is that a function must always return a value, but a procedure may or may not return a value.

3.2 Anti–random technique

In this section, we will apply anti-random testing technique to generate testing data for Oracle named blocks in order to achieve branch coverage with minimum number of test cases. e use all test vectors from testing domain to build the required anti-random sequence of testing. The maximum distance condition means, picking new test vector from the remaining vectors in the testing domain, to develop testing sequence, which should satisfy the criteria that the picked one from the domain should be the most differed vector from already picked ones in the sequence. The procedure of constructing the maximum distance anti-random test sequence is summarized in the following steps: Step 1: Assign an arbitrarily chosen value to obtain the first test vector. Step 2: To obtain each new vector, evaluate the total distance for each of the remaining combinations, using Hamming or Cartesian measures, with respect to the combinations already chosen. Then choose one that gives the maximal distance to add it to the set of selected vectors. Step 3: Repeat Step 2 until all combinations have been used, or until the desired numbers of test cases have been generated.

3.3 White-box anti-random testing White-box anti-random testing algorithm is based on two basic assumptions: the first one, where in every condition statement in the code there is a value or values for program parameters that will execute the condition statement with true or false result. The second one, any input values with small distance between them, will most likely result same output. Following are the steps of using anti-random technique in white box testing as our algorithm: Step 1: Analyzing program parameters This is done by analyzing the number of parameters, parameter types and parameter data types. Step 2: Analyzing program structure In this step, we list all condition branches and the relation between them. We check the condition result of the program execution without knowing the details of that condition

statement. We add an insert statement for this condition into log table for the program execution with false result, and in the true region of this condition we add update statement to set the result to true. So, to achieve branch coverage of this condition we should have at least two records, one with true result and the other with false. Step 3: Generating testing data In this step, each parameter domain is divided into sub-domains, and each sub-domain is presented as single element for that sub-domain. General testing means small numbers of testing sub-domains, each representing large volume of data. Consequently, detailed or mass testing means large number of sub-domains, each representing small volume of data. To make sure that we have same volume for each sub-domain, we split the main domain for each parameter into 2 to the power of testing level. As an example, if we have a number parameter with expected domain from 1 to 100,000 and we set testing level equal to 4 to test the program, then we have sub-domains which are equal to 16, each having 6250 elements, as shown in Figure 3.

Figure 3. Splitting domain

However, data generation can be summarized as follows: we choose a parameter of the program in random, as a starter parameter, and then we split the domain for that parameter into sub-domains based on the testing level as input from the tester; noting that we pick a random element from each sub-domain. After that we sort subdomains in anti-random sequence using Hamming or Cartesian distance measure as testing parameter. Figure 4 shows sorted data domain anti-randomly, where the top sequence is normal sequence of testing, and the bottom one is the sequence after sorting anti-randomly.

Figure 4. Sorted domain anti-randomly

Step 4: Executing program with generated test cases This step is done by executing the program using the generated data in the sorted sequence done in step three, and after each run we check if the branch coverage is achieved or not by checking the log table. Test cases are grouped into number of groups, then starting the execution for each group in parallel. Figure 5 shows the sequence execution of test cases done by a single worker. Figure 6 shows the structure of parallel execution by three workers, and a single worker dedicated to check the coverage status.

Figure 5. Single worker execute test cases sequentially

Figure 6. Parallel execution done by several workers

3.4 White-box anti-random testing contributions and limitations This algorithm is easy to implement, easy to use, and it covers input data domain in fast way, where it prevents local maxima and local minima problems, which is found in Genetic Algorithms. Moreover, by using this algorithm there is no need to know the details of operators and operands for each condition in order to generate useful test cases. Feedback from the conditions taken in as fitness function is also not needed. Finally, endless execution and folding problem is prohibited in this algorithm by using expansion parameter concept and controlling testing level by user. In the contrast, high computation process is the main

cons of this algorithm, especially with large number of input parameters and deep testing level. Furthermore, we face a big issue of computing distance in character parameter domain, which needs to be addressed, besides the well known problems of defining proper values of testing level and number of workers. At last, Mislaid of fitness function in our algorithm is considered as a weakness, because of no feedback from execution, which may help to get the desired solution.

4. Experiments and results In this experiments, the authors generated test cases based on program's parameters and defined testing level. Then they sorted test cases anti-randomly based on distance measurement's parameter, and finally executed the programs using the generated test cases in its anti-random sequence, where parallel execution can be done based on the number of workers' parameter. We used ten different Oracle programs to test our algorithm with different testing level values, different distance measurements and appropriate number of workers.

4.1 Anti-random executer They have demonstrated anti-random executer as an application, which is used to implement our testing algorithm. It is an Oracle client-server application constructed from two parts, where the first part is the database side, which contains all working tables, and the second part is the client side or interface part, which is used by the tester to execute the testing process, where it is simple and easy to use to test any Oracle program unit.

4.2 Setup parameters and evaluation metrics The parameters in our algorithm are categorized in two groups. In the first group, each parameter takes one value for all executions, where this value is selected based on many executions until it is selected as an optimal value. Examples of parameters in this group are number of workers value, expected domain value and the period of checking branch coverage. The number of workers is the number of processes which are responsible to execute ready test cases. Expected domain is not a regular stand alone parameter, its value selection depends on the nature of the target program and the usage of that parameter for it. In the second group, each parameter takes many different values, one for each execution record, such as testing level value and distance measurement (Hamming distance or Cartesian distance). Testing level is the parameter which determines the number of generated test cases of a certain execution, and distance measurement is the parameter which determines the approach of sorting test cases antirandomly by using Hamming distance and Cartesian distance.

The performance of anti-random testing is assessed by testing total running time that includes test cases generation time, test cases sorting time and test cases execution time. The performance assessed by our algorithm target, is branch coverage. Thus, we run each of ten programs six times each, using Hamming distance and Cartesian distance.

4.3 Experimental results In this section, the authors discussed the executions of white-box anti-random testing on number of Oracle programs, in order to evaluate the result of such technique. They applied the proposed algorithm on ten oracle programs. They also summarized these programs in Table 1.

Table 1. Programs under test

To apply the proposed algorithm on oracle programs, we did minor manual pre-processing phase to prepare each program to be tested; this preprocessing was done for each program separately, and did not affect program logic, execution behavior, and program result. This phase consist of two steps; first one is studying program logic to extract expected domain values and the second step is adding “commit” oracle command at the end of each program as prerequisite condition to run our proposed algorithm. They run each program six times with different execution parameters and the result of branch coverage and execution time was computed and observed.

4.3.1 Program 1: Bonus program This program has two while-statements that are nested inside an IF statement in which all conditions are simple. Also, in this program there are three conditional statements, the first one includes compound conditions; both conditions present a parameter with equal

operator (=) to specific value, which is some how hard to be achieved as true branch. When true branch is achieved, two nested while-statements using different parameters has to be achieved. However, it was easier due to greater than (>) and less than (” and less than operator “