International Journal of Computers and Applications, Vol. 31, No. 4, 2009
SOFTWARE LIBRARY USAGE PATTERN EXTRACTION USING A SOFTWARE MODEL CHECKER C. Liu,∗ E. Ye,∗ and D.J. Richardson∗∗
Abstract
written to after it is closed, which means that file writing function cannot be invoked after file close function is invoked. Software library rules are typically only stated implicitly and informally in the API documentation. For instance, as stated in technical documentation on the Microsoft Developer Network (MSDN), the socket class in the .NET Framework Class Library has the following rule: “When you are finished sending and receiving data, use the Shutdown method to disable the Socket. After calling Shutdown, call the Close method to release all resources associated with the Socket”.1 Therefore, programmers are prone to ignore these rules, and thus incorrectly use API functions. Such incorrect usage is hard to detect, because in the current state of practice, these implicit rules cannot be checked by a compiler or other software tools. With the help of model checking techniques [1], API usage rules can be specified as temporal properties that model checking tools can recognize, thus programmers can use tools to verify if programs using the API violate the rules [2, 3]. Model checking is an automatic method for formally verifying finite-state concurrent systems. The process of model checking a finite-state concurrent system works as follows: the system is converted into a formal model, usually a state-transition graph; the specification of the system’s behaviour is usually described as temporal logic formulas; an efficient search procedure is used to determine whether or not the specification is satisfied in the model. Model checking has been used successfully in practice to verify complex sequential circuit designs and communication protocols. Due to the difficulty of producing correct software, there has been increasing interest in the application of the model checking technique to software. There are various software model checkers available, such as SPIN [3], SLAM [2], BLAST [4], JavaPathFinder [5], Bandera [6], and Boger [7]. SPIN [3] is an on-the-fly linear temporal logic (LTL) model checker based on an automata-theoretic verification method, which is used to formally verify models of distributed software systems. SLAM [2] aims to automatically check that a C program correctly uses an API
The need to manually specify temporal properties of software systems is a major barrier to wider adoption of software model checking.
To address this problem, we propose to automatically
extract software library usage patterns, i.e., one type of temporal specifications. Our approach uses a model checker to check a set of template-based pattern candidates against existing programs, and identifies valid patterns based on model checking results. We applied our approach to extract one valid pattern of the C standard library from 500 C programs in Sourceforge.net using BLAST, and used the extracted pattern to detect an error in an open source project.
Key Words Software verification, model checking, software library usage pattern
1. Introduction Software programmers often develop software systems by using existing software libraries, as these libraries usually implement the functionalities needed by software systems with well-defined application programming interfaces (API). A library API consists of a set of library functions, through which library features can be accessed. Implicit constraints often exist among API functions of the same library, and software systems must obey these constraints to ensure proper execution of these API functions. We refer to these constraints as software library usage rules, or API usage rules. For example, a library that provides file operations may include three API functions: the first API function opens a file and returns the file handle, the second API function closes the file given a file handler, and the third API function writes some text to the file given a file handler. An API usage rule may require that a file be opened before text can be written to it, which means that the file open function should be invoked before file writing function is invoked. Another rule is that a file cannot be ∗
Ohio University, Athens, Ohio 45701; e-mail: {liuc, ey171304}@ohio.edu ∗∗ University of California, Irvine, California 92697; e-mail:
[email protected] Recommended by Dr. Letha Etzkorn (paper no. 202-2504)
1
247
Socket class: http://msdn.microsoft.com/library/en-us/cpref/ html/frlrfsystemnetsocketssocketclasstopic.asp.
refers to a complete program (including a single file that can execute by itself) that uses the API of that library; has been tested, deployed, and used; and is generally known as not containing unacceptable defects. The key idea of our approach is to take known good programs using the same library as the oracle, use a model checker to verify API usage pattern candidates against them, and extract valid patterns from the pattern candidates based on the model checking result. These valid patterns can help programmers learn about correct software library usage. They can also be used to verify new programs using the same model checking process. As the API usage patterns are automatically extracted, our approach can relieve the programmers from the burden of writing specifications. Programmers can thus focus on developing software systems, and then apply software model checkers to verify them, which will in turn enhance the practicality of software model checking. Software library usage rules represent intrinsic relationships among API functions of a software library. They may be stated explicitly or implicitly; they may be specified formally or informally. Nevertheless, they always exist. Our observation is that software library usage patterns extracted by verifying pre-constructed usage pattern candidates against a large number of known good programs can closely reflect library semantics, because they are at least obeyed by known good programs. The rest of the paper is organized as follows. Section 2 describes the details of our approach and includes an illustrative example. Section 3 presents the experiment result of applying our approach to C programs using the Open Secure Sockets Layer (OpenSSL) library and the C standard library. Section 4 discuses the limitations of our current tool implementation. Section 5 discusses related work. Section 6 concludes the paper.
according to the API usage rules. Given a safety property to check on a C program, the SLAM process iteratively creates, analyses, and refines a boolean abstraction of the program until the program is validated. BLAST [4] is also an automatic model checker for checking safety properties of C programs. It implements an “abstract—model check—refine” loop to check for reachability of a specified label in the program, and integrates the three steps tightly through a “lazy abstraction” algorithm. JavaPathFinder [5] is an explicit-state model checker for Java programs. It integrates verification, program analysis, and testing. It works at the bytecode level, and checks all deadlocks and unhandled exceptions (e.g., assertion errors). Boger [7] is an extensible model checking framework designed to support both general purpose and domain-specific software model checking. It can be used to check properties of various modern software artifacts, to build new domain-specific engine, and to teach model checking concepts. Bandera [6] is an integrated collection of program analysis, transformation, and visualization components. It extracts finite-state models from Java source code based on the property to be checked, and checks the model using existing model checkers such as Bogor. Software model checking is a promising approach to improve software quality because it can search all possible execution paths for specific errors. However, to use software model checking, programmers needs to manually specify temporal properties of software systems. Temporal properties are typically specified in temporal logics such as LTL, which are often difficult to understand. In [8], Holzmann pointed out that it is difficult to define a simple temporal property using LTL. Dwyer et al. presented an example of a complicated LTL formula in [9] to illustrate the difficulty of composing and understanding LTL formulas. Although SLAM and BLAST use automata-based specification, which is easier to understand than temporal logics for programmers, it is slightly different from the implementation languages of software systems. Programmers still need to spend extra time to learn them. Moreover, temporal properties are usually defined based on the model abstracted from the software system. As a result, to define a temporal property, programmers must not only be familiar with the specification language but also understand well the structure of the model, not just the software system itself. Therefore, the specification of temporal properties is a difficult, time-consuming, and error-prone process. Many programmers are reluctant to construct temporal specifications; some are unable to specify correct specifications. Thus, it becomes a major barrier to wider adoption of software model checking. To solve this problem, we propose to automatically extract temporal specifications representing API usage patterns from known good programs based on model checking. API usage patterns or software library usage patterns represent common ways in which most programmers use API functions in a library. Although they cannot be as strict as API usage rules that programmers cannot violate (otherwise some error would occur), following these patterns can guarantee that programmers do not violate API usage rules in most of cases. A known good program of a library
2. Approach Our approach mainly consists of two steps: API usage pattern template instantiation and API usage pattern candidate verification. It predefines some API usage pattern templates. API usage pattern candidates are generated by instantiating pattern templates with concrete API functions. A model checker is used to verify the pattern candidates against known good programs, and extract valid patterns from the pattern candidates based on the model checking results. The process of our approach is shown in the dataflow diagram in Fig. 1, in which rectangles represent data and ovals represent data processing. To use our approach, programmers only need to provide a group of related API functions and known good programs using those API functions as input, and then get the valid patterns as the output. Related API functions are a group of API functions provided by the same library that may have an impact on each other. These functions are often invoked together in a sequence to complete a task. Programmers are usually interested in only usage rules among these API functions. Before introducing the details of our approach, we first introduce the API usage pattern templates and the model checkers used in our approach. 248
• The function-pair pattern implies the strictalternation pattern. • The strict-alternation pattern implies the push-pop pattern. 2.2 Model Checker Our approach can extract six categories of API patterns from known good programs if the programs and the API patterns can be converted to a form understandable by a model checker. For our approach to be helpful, the conversion should be done automatically. Therefore, a suitable model checker should be able to directly accept source code, and the specification language of the model checker should be able to express the six categories of patterns. In this paper, our approach is focused on extracting API patterns from C programs. We choose BLAST as the model checker because BLAST’s specification language [10] is automata-based specification language, which define a state machine that monitors the execution of behaviours of the program at the API function level. Therefore, it is straightforward to use them to represent patterns related with API function orders. Because BLAST completely separate specification input from model input, specification templates for six API patterns can be easily created. Six categories of API usage pattern templates are represented in the BLAST specification language, among which the push-pop pattern is shown in Fig. 2. In those templates, the keyword global defines a global variable (i.e., state); the keyword event is used to change global state and verify properties based on the execution of a C program; the keyword pattern specifies which API function activates an event; the keyword guard specifies the checks to be made before taking any actions if the pattern is matched; the keyword action specifies actions to be taken at certain points during execution after the guard condition is satisfied.2 For example, the first event in Fig. 2 specifies that if an API function named funcA is invoked in the program to be verified, then the value of global variable number should be bigger than or equal to zero. If not, the specified property is violated; otherwise, the value of global variable number should be added one. Note that in those templates, funcA and funcB are symbolic names for API functions. They will be replaced by concrete API function names later when usage pattern candidates are generated.
Figure 1. Automatic API usage pattern extraction process based on model checking. 2.1 API Usage Pattern Templates Currently, our approach focuses on extracting six categories of API usage patterns as follows. We will add more categories as the project evolves. 1. The initialization pattern: One API function is always invoked first among all the API functions in the group, such as initVerify() (or initSign()) in the java.security.Signature class in the SUN’s standard Java class library. 2. The finalization pattern: One API function is always invoked last among all the API functions in the group, such as the Close() in the Socket Class in the .NET Framework Class Library. 3. The push-pop pattern: At any given point, one API function is always invoked at least as many times as compared to the other, such as push() and pop() in a stack library. The invocation times of each API function refers to the number of times it actually gets executed in the program. 4. The strict-alternation pattern: Two API functions are always invoked in strict alternation, such as KeAcquireSpinLock() and KeReleaseSpinlock() in Windows Driver Development Kit. 5. The function-pair pattern: Two API functions are always invoked equal times and also invoked in strict alternation, such as ZwCreateFile() and ZwClose() in Windows Driver Development Kit. 6. The adjoining-function pattern: Two API functions are always invoked together in the same order, and no other API functions can be invoked between them, such as the Shutdown() and Close() in the Socket Class in the .NET Framework Class Library.
2.3 Steps The detailed steps of our approach are listed below: 1. API usage pattern template instantiation: For all the related API functions in the group, a list of API usage pattern candidates is generated by replacing the symbolic function names in the pattern templates with concrete API function names. The pattern candidates represent all possible relationships among API functions in the group.
Among these six categories, the first two categories deal with one API function while the other four categories deal with two API functions. The last four categories are related in the following ways: • The adjoining-function pattern implies the functionpair pattern, which means that any two API functions satisfy the function-pair pattern if they satisfy the adjoining-function pattern, whereas the reverse may not be true.
2
249
BLAST’s user manual: blast/.
http://mtc.epfl.ch/software-tools/
API functions used in three known good programs of this library, as shown in Figs. 4–6. The details of the process are listed below:
global int number = 0; event { pattern { $? = funcA;} guard { number >= 0} action { number = number + 1;} } event { pattern { $? = funcB;} guard { number > 0} action { number = number − 1;} }
1. Suppose that the programmer identifies a group of four related API functions: initialize(), push(), pop(), isEmpty() and three known good C programs stack1, stack2, stack3. There are four (P14 ) pattern candidates for the initialization pattern and finalization pattern respectively (i.e., one pattern for each API function), and twelve (P24 ) pattern candidates for each of the other four patterns (i.e., all the possible permutations of choosing two API functions from four API functions). 2. Our approach generates concrete pattern specifications from the specification templates by replacing the symbolic function names in the templates with concrete API functions. One of the concrete specifications for push-pop pattern is shown in Fig. 7.
Figure 2. Specification template for the push-pop pattern in the BLAST specification language. 2. API usage pattern candidate verification: A model checker (i.e., BLAST) is used to check all the pattern candidates against all known good programs to extract valid patterns. The pattern candidates and the programs are used as the input for the model checker during verification. The model checker checks one pattern candidate against one program every time. A pattern is valid only if it is satisfied by all known good programs. If we cannot get a number of known good programs (i.e., we cannot examine the correctness of sample programs), we can identify valid API usage patterns according to the ratio of the number of the “good” programs (by which an API usage pattern was satisfied) to the number of all the programs in the experiment. Suppose we represent the number of the programs by which an API usage pattern was satisfied as NS, the number of the programs by which an API usage pattern was violated as NV, we can calculate the ratio T by the formula T = N S/(N S + N V ). Moreover, as BLAST gives a counterexample for a program by which an API usage pattern is violated, we can try to find the real error in the programs by which the valid patterns were violated according to the counterexamples.
3. Based on the three known good stack application programs, stack1, stack2, and stack3, we conclude through model checking that two pattern candidates are valid. They are “initialize() should be invoked first” and “push() should be invoked equal times with or more times than pop()”. #include “stack.h” int main() { int x = 1; initialize(); push(x); pop(&x); push(x); push(x); pop(&x); exit(0); }
2.4 An Illustrative Example
#include “stack.h” int main() { int x = 1; initialize(); while (!isEmpty()) pop(&x); push(2); if (!isEmpty()) pop(&x); else push(x); exit(0); }
#include “stack.h” int main() { int x = 1; initialize(); push(x); while (!isEmpty()) pop(&x); push(2); if (!isEmpty()) pop(&x); else push(x); exit(0); }
To illustrate our approach, we take a C library that implements a stack (shown in Fig. 3) as an example, and use our approach to extract the usage patterns of the stack library
const int SSize = 8; static int data[8]; static int top; void initialize(void) { . . . } int push(int d) { . . . } int pop(int *d) { . . . } int topValue(int *d) { . . . } int isEmpty(){ . . . } int getSize(){ . . . }
Figures 4–6. Source code of a C program stack1, stack2, and stack3.
Figure 3. Snippet of a C library stack.h. 250
API libraries. We selected the OpenSSL library3 and the C standard library for the experiments. OpenSSL is an open source cryptography library implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols. In the C standard library, we focus on API functions dealing with file operations. The experiments were performed on a Gateway E-2000 PC with a 2.4 GHz Pentium 4 CPU and 512 MB RAM. It runs the Fedora 4 Linux operating system.
global int number = 0; event { pattern { $? = push($?);} guard { number >= 0} action { number = number + 1;} } event { pattern { $? = pop($?);} guard { number > 0} action { number = number − 1;} }
3.1 The OpenSSL Library For the OpenSSL library, we selected a group of six API functions, all of which deal with the SSL/TLS sessions defined in the SSL_SESSION structure. They are SSL_free(), SSL_new()}, SSL_set_fd(), SSL_connect(), SSL_write(), and SSL_read(). We also selected four applications using the OpenSSL library: CashCow,4 Slush,5 PayPal Sender,6 and Netcat SSL.7 These four OpenSSL applications have been released for more than three years and used by many people. Therefore, they are relatively stable and can serve as known good programs. The process of using our approach to extract patterns from the four OpenSSL applications is shown in Fig. 8. From the four OpenSSL applications, it took 2 min and 56 sec to extract four valid OpenSSL API usage patterns
Figure 7. Concrete specification for the push-pop pattern in the BLAST specification language. In this example, we successfully extracted two valid patterns from the three known good programs using BLAST in 23 s. 3. Experiment To validate our approach, we developed an automated usage pattern extraction tool named LtRules (pronounced “light rules”) [11, 12]. It was implemented as an Eclipse plug-in with a Python usage pattern extraction engine. We used LtRules to perform experiments on real-world
Figure 8. The automatic model-checking-based API usage pattern extraction process for OpenSSL applications. 3 4 5 6 7
251
OpenSSL: http://www.openssl.org/. Cashcow: http://www.cashcow.dk/. Slush: http://violet.ibs.com.au/slush/. Paypal sender: http://members.chello.se/hampasfirma/ppsend/. Netcat SSL: http://sourceforge.net/projects/nssl/.
out of 132 (2*P16 + 4*P26 ) pattern candidates, which represent all possible relationships between six OpenSSL API functions. The statistics of the experiment result is shown in Figs. 9–11. Each figure shows a table with the pattern candidates as the rows and the known good programs as columns. The value of each cell could be s, v, or f. The code s stands for “Specification (i.e., API usage pattern) satisfied”. All cells with s are in red, or the darkest colour in the figure. The code v stands for “specification Violated”. All v cells are in white. The code f stands for “Fatal error occurred in the verification process”, which means that BLAST cannot tell whether the specification is satisfied or violated by the program under verification. All f cells are in blue, or the second darkest colour in the figure. Therefore, if a row has all s, then the pattern in that row is satisfied by all programs, which show up as a red horizontal line in the figure. In any given row, the more cells are in red, the more likely the pattern at that line is a valid pattern. The four valid patterns are listed below: 1. push-pop[SSL_new(), SSL_free()], which represents the push-pop pattern between SSL_new() and SSL_free(). The following patterns are presented in the same format. 2. push-pop[SSL_new(), SSL_set_fd()]. 3. push-pop[SSL_new(), SSL_connect()]. 4. strict-alternation[SSL_set_fd(), SSL_connect()]. All four valid API usage rules indicate common usages of these OpenSSL library functions and reflect their inherent semantics. For example, the first rule states that the SSL_new() function, which creates a new SSL structure for use with an SSL session, and the SSL_free() function, which frees an allocated SSL structure, satisfy the push-pop rule, i.e., at any given point in program execution, SSL_free() is always invoked equal or less times than SSL_new(). It is clear from the semantics of these two functions that if SSL_free() is invoked more times than SSL_new(), it would attempt to free SSL structures already released, and therefore is inconsistent with its original semantics. As another example, the last rule reflects that the SSL_set_fd() function, which assigns a socket to an SSL structure, and the SSL_connect() function, which starts an SSL session with a remote server application, are commonly used in strict alternation.
(ANTLR) to output the name of all the API functions invoked in a C file. According to the statistics result of every C function’s occurrence times from 4696 C files that can be processed by the modified parser, we found that API functions dealing with file operations were invoked most often. Thus we chose a group of 14 API functions related with file operations: fprintf(), fopen(), fclose(), fputs(), fwrite(), fgets(), fread(), fflush(), fputc(), fseek(), feof(), fgetc(), fscanf(), and ftell(). After choosing the API function group, we further filtered out 1058 C files that include at least 2 API functions among 14 API functions for this experiment. Then we chose the first 500 C files, almost half of all the 1058 files, for the experiment. Due to the number of sample programs, we did not check the maturity of these programs. We used all of them directly in the experiment and identified API usage patterns from the experiment result by setting the threshold value of the ratio T to 0.9. This experiment took more than one day. Part of the experiment result is shown in Fig. 12 (due to the page size limitation, only the result of 30 pattern candidates and 30 C files were included in the figure). The row, the column, and the value of each cell have the same meaning as in Fig. 9. There is one additional code in this figure. The code n stands for “No new predicate found in the verification”, which means that BLAST cannot tell whether the specification is satisfied or violated by the program under verification. All n cells are in green, or the third darkest colour in the figure. With a threshold value of 0.9, we identified only one valid API usage pattern, which is the push-pop rule between fopen() and fclose(). It was satisfied by 208 programs, and violated by 14 programs. The ratio T is 218/(218 + 14) = 0.93, which is greater than 0.9. After identifying the valid API usage pattern, we looked into the seven counterexamples given by BLAST for the seven programs among the fourteen programs by which the push-pop pattern between fopen() and fclose() was violated. We found there was a real error in one program, which is kino2mjpeg.c in the Kino project8 (Its path is kino/kino/src/kino2mjpeg.c.). In its main() function (shown in Fig. 13), the WAVStruct_Init() function invokes fopen() while WAVStruct_Close() invokes fclose(). According to the counterexample given by BLAST, WAVStruct_Close() can be invoked without WAVStruct_Init() being invoked first, in other words, fclose() can be invoked without fopen() being invoked first. In this case, fclose() would try to close a NULL file token, and the program would crash. Among the other six programs, four attempt to close stdin or stdout, which is an unnecessary operation. These four programs are fon2fnt.c in FarsiTeX project,9 minigzip.c in okvm project,10 untar.c in FreeDOS project,11 and utmpdump.c in Morphix project.12 From the code snippet of the main() function in fon2fnt.c in
3.2 The C Standard Library For the C standard library, we selected sample programs from open source software projects hosted by SourceForge.net. We checked out all the C projects in SourceForge.net. From all the 450774 C source files included in these projects, we filtered out 10528 C files that include the main() function, and can be directly processed by BLAST without complaints. After the removal of duplicate files, and files that will make BLAST stuck in the verification process (i.e., BLAST keeps running without giving a result), there were 6803 C files left. As we planned to select a group of API functions that were invoked most often among these C files, we modified the parser ANother Tool for Language Recognition
8 9 10 11 12
252
Kino: http://sourceforge.net/projects/kino/. FarsiTeX: http://sourceforge.net/projects/farsitex/. okvm: http://sourceforge.net/projects/okvm/. FreeDOS: http://sourceforge.net/projects/freedos/. Morphix: http://sourceforge.net/projects/morphix/.
Figure 9. The result of the experiment on extracting API usage patterns of the OpenSSL library from four applications: part 1 of 3. 253
Figure 10. The result of the experiment on extracting API usage patterns of the OpenSSL library from four applications: part 2 of 3. 254
Figure 11. The result of the experiment on extracting API usage patterns of the OpenSSL library from four applications: part 3 of 3.
actually correct programs, which are volx.c in Sarien project,13 and chotplay.c in gPhoto project.14 They were regarded as violation because BLAST regards the exit()
FarsiTeX project (shown in Fig. 14), we can see that if argc equals to 1, the variable fon will be set to stdin, and the variable fnt will be set to stdout. Then fclose() will be invoked twice to close fon and fnt in the end, which are unnecessary operations. The other two programs are
13 14
255
Sarien: http://sourceforge.net/projects/sarien/. gPhoto: http://sourceforge.net/projects/gphoto/.
Figure 12. Part of the experiment result on extracting API usage patterns of the C standard library from programs hosted by SourceForge.net. int main(int argc, char * argv[]) { ... switch (argc) { case 1: fon = stdin; fnt = stdout; ... } fclose(fon); fclose(fnt); ... }
int main(int argc,char * argv[]) { ... while (fgets(. . .)){ if . . . else if (!strncmp(. . .,“OPEN WAV FILE”,. . .)) wav = WAVStruct Init(. . .); else if (!strncmp(. . .,“CLOSE AUDIO”,. . .)) WAVStruct Close(wav); else if . . . } ... } of
Figure 14. Code snippet of function main() of fon2fnt.c in the FarsiTeX project.
function as a normal function and continues analysing the program after exit() is invoked.
patterns). Programmers can learn about common software library usage pattern based on the extracted API patterns. They can also use these patterns to check new unverified programs through the same model checking process. There are two issues that one must pay attention to before adopting our approach: 1. The quality of the extracted patterns depends on the quantity and quality of the existing programs. If the number of programs or the number of API function invocations is too small, the extracted patterns will
Figure 13. Code snippet of function kino2mjpeg.c in the Kino project.
main()
4. Discussion These two experiments demonstrated that our approach extracted valid API usage patterns from existing programs in a reasonable amount of time (for the C standard library experiment, actually we can reduce the experiment time by utilizing the implications that exist among the last four 256
running the instrumented program over a set of test cases, and testing all the possible invariants against the values of the instrumented variables captured from program executions. Hangal and Lam proposed an approach to extract likely invariants by starting with the strictest invariant candidates, and continually relaxing them based on the running results of the instrumented programs. They use the extracted invariants to detect software bugs and even their root causes [15]. The distinction between these two techniques and ours is that they deal with the value relationships among a program’s variables, while ours focuses on a program’s temporal properties such as the execution sequences of functions. Their approach focuses on data flows, while ours focuses on control flows.
inevitably include some meaningless ones. 2. The number of model checker invocations will increase very quickly with the increase of the number of API functions because our approach deals with all the possible relationship combinations among API functions, and verifies one API function candidate at a time using a model checker. We can reduce the number of model checker invocations by utilizing the implications that exist among the last four patterns. After verifying all the push-pop pattern candidates against sample programs, only the strict-alternation pattern candidates whose corresponding push-pop patterns are valid need to be chosen for verification, as the strict-alternation pattern implies the push-pop pattern. The quantity and quality of API usage pattern extraction is also affected by the limitations of the underlying model checker. First, usage patterns that the tool can extract are limited by the expressiveness of the model checker’s specification language. Second, currently, the underlying model checker imposes some restrictions on programs-under-verification. For example, BLAST may give incorrect verification results when data pointers are involved, and it does not support function pointers. It also regards the exit() function as a normal function and continues analysing programs after exit() is invoked. All these limitations may cause our approach to miss good patterns or extract bad patterns. These limitations can be removed by using a more powerful model checker when it becomes available.
5.3 Specification Extraction Many researchers have also made attempts to automatically extract software temporal properties from software systems [16–21]. Ammons et al. [17] proposed an approach to infer from dynamic traces frequent method sequences that capture both temporal and data dependencies. Whaley et al. [20] proposed a static analysis technique and a dynamic instrumentation technique to automatically extract from existing code of the behavioural interface of a Java class. Alur et al. [16] proposed a synthesis approach to automatically extract from program behavioural interface for Java classes, which specifies the correct sequencing of methods calls. Engler et al. [18] proposed techniques that automatically extract correctness rules such as API function-pair from program statically by tailoring rule templates to it. Weimer & Necula [19] proposed an approach that uses information about error handling to learn temporal safety rules automatically. Yang et al. [21] proposed an approximate dynamic inference algorithm to mine from the imperfect traces temporal API rules, especially alternation rule between two API functions. They then used contextual information (e.g., this object) and three heuristics to filter improper rules. Different from existing specification extraction techniques [17–21], our approach uses a model checker to verify pattern candidates against sample programs. Due to the exhaustive nature of model checking, the patterns extracted by our approach are guaranteed to be consistent with sample programs, while those of specification extraction techniques cannot guarantee that (with the exception of the technique in [16]). Although the specification extraction technique in [16] can extract specification consistent with sample programs, it requires more user inputs than our approach: it needs from the user the exception predicate, an initial set of predicates, and a specification size besides the program, while our approach only needs a group of related API functions and known good programs as input.
5. Related Work 5.1 SLAM The SLAM [2] project developed at Microsoft Research aims at automatically checking that a C program correctly uses the API according to API usage patterns. In the SLAM project, Ball et al. use a novel approach to automatically create environment models (i.e., models of the kernel) via training [13]. The idea of training is to create abstractions of the API procedures by model checking several programs that use a common API. Our approach shares some common features with the training approach. Our approach also uses known good programs for extraction, and needs the help of a model checker. However, our approach deals with interrelationship between the API procedures, whereas the training approach focuses on API procedures themselves. Therefore, these two approaches are complementary and can be used together. The API usage patterns extracted by our approach can be represented in SLIC, which could alleviate SLAM’s limitation of requiring manually written API usage patterns. 5.2 Program Invariants Ernst et al. proposed an approach to automatically learn likely invariants involving program variables from dynamic traces [14]. The approach infers invariants by instrumenting the source program to trace the variables of interest,
6. Conclusion We have successfully applied the model checking technique to extract software library usage patterns from existing 257
C programs. We chose BLAST as the C model checker for our approach, and developed an Eclipse plug-in to generate pattern candidates and invoke BLAST to check them automatically. We used C programs in the SourceForge for experiment, and extracted a valid API usage pattern. Furthermore, we used the extracted pattern to find an error in a C program and unnecessary operations in four C programs. Our approach turns informal API usage patterns into formal ones that software model checkers recognize, so that programmers can use model checkers to verify whether software systems violate the API usage patterns. Meanwhile, our approach automatically extracts API usage patterns, which makes implicit API usage patterns explicit. Unlike natural language documentation, these automatically extracted usage patterns are guaranteed to be consistent with code. More importantly, our approach eliminates the need for programmers to manually write usage pattern specifications if they wish to perform software model checking. We expect this approach to have a positive impact on wider adoption of software model checkers. In the future, we plan to lower the overhead of our approach by looking for more ways to reduce the number of model checker invocations so that our approach can be scaled to large programs. We also intend to integrate our approach with other model checkers and extract API usage rules from programs in other languages such as Java to further validate our approach. Finally, we wish we could express more complicated usage patterns using the specification languages of current model checkers.
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
References [1] E.M. Clarke, O. Grumberg & D.A. Peled, Model checking (Cambridge, MA: MIT Press, 1999). [2] T. Ball & S.K. Rajamani, The SLAM project: debugging system software via static analysis, in POPL ’02: Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (New York, NY: ACM Press, 2002), 1–3. [3] G.J. Holzmann, The model checker SPIN, IEEE Transactions on Software Engineering, 23(5), 1997, 279–295. [4] D. Beyer, T.A. Henzinger, R. Jhala, & R. Majumdar, The software model checker BLAST: applications to software engineering, International Journal on Software Tools for Technology Transfer, 9(5), 2007, 505–525. [5] W. Visser, K. Havelund, G. Brat, S. Park, & F. Lerda, Model checking programs, Automated Software Engineering Journal, 10(2), 2003, 203–232. [6] J.C. Corbett, M.B. Dwyer, J. Hatcliff, S. Laubach, C.S. Pasareanu, Robby, & H. Zheng, Bandera: extracting finitestate models from Java source code, in ICSE ’00: Proceedings of the 22nd International Conference on Software Engineering (NY USA: ACM Press, 2000), 439–448. [7] Robby, M.B. Dwyer, & J. Hatcliff, Bogor: an extensible and highly-modular software model checking framework, in ESEC/FSE 2003: Proceedings of the 4th Joint Meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering, New York, NY, (2003), 267–276. [8] G.J. Holzmann, The logic of bugs, in SIGSOFT ’02/FSE10: Proceedings of the 10th ACM SIGSOFT Symposium on Foundations of Software Engineering (NY, USA: ACM Press, 2002), 81–87. [9] M.B. Dwyer, G.S. Avrunin, & J.C. Corbett, Patterns in property specifications for finite-state verification, in ICSE ’99: Proceedings of the 21st International Conference on Software
[19]
[20]
[21]
Engineering (Los Alamitos, CA, USA: IEEE Computer Society Press, 1999), 411–420. D. Beyer, A.J. Chlipala, T.A. Henzinger, R. Jhala, & R. Majumdar, The BLAST query language for software verification, in SAS ’04: Proceedings of the 11th International Static Analysis Symposium (Berlin, Heidelber: Springer-Verlag, 2004), 2–18. C. Liu, E. Ye, & D.J. Richardson, LtRules: an automated software library usage rule extraction tool. In ICSE ’06: Proceedings of the 28th International Conference on Software Engineering, Research Demonstrations, New York, NY, (2006). C. Liu, E. Ye, & D.J. Richardson, Software library usage pattern extraction using a software model checker, in ASE ’06: Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering, Washington, DC, (2006), 301–304. T. Ball, V. Levin, & F. Xie, Automatic creation of environment models via training, in TACAS ’04: Proceedings of the 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (2004), 93–107. M.D. Ernst, J. Cockrell, W.G. Griswold, & D. Notkin, Dynamically discovering likely program invariants to support program evolution, IEEE Transactions on Software Engineering, 27(2) February 2001, 1–25. S. Hangal & M.S. Lam, Tracking down software bugs using automatic anomaly detection, in ICSE ’02: Proceedings of the 24th International Conference on Software Engineering (NY, USA: ACM Press, 2002), 291–301. R. Alur, P. Cerny, P. Madhusudan, & W. Nam, Synthesis of interface specifications for Java classes, in POPL ’05: Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles Of Programming Languages (ACM Press, 2005), 98–109. G. Ammons, R. Bodik, & J.R. Larus, Mining specifications, in POPL ’02: Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (New York, NY: ACM Press, 2002), 4–16. D. Engler, D.Y. Chen, S. Hallem, A. Chou, & B. Chelf, Bugs as deviant behavior: a general approach to inferring errors in systems code, in SOSP ’01: Proceedings of the 18th ACM Symposium on Operating Systems Principles (New York, NY: ACM Press, 2001), 57–72. W. Weimer & G. Necula, Mining temporal specifications for error detection, in TACAS ’05: Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (2005). J. Whaley, M.C. Martin, & M.S. Lam, Automatic extraction of object-oriented component interfaces, in ISSTA ’02: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis (ACM Press, 2002), 218–228. J. Yang, D. Evans, D. Bhardwaj, T. Bhat, & M. Das, Perracotta: mining temporal API rules from imperfect traces, in ICSE ’06: Proceedings of the 28th International Conference on Software Engineering, New York, NY, (2006).
Biographies Chang Liu is an associate professor of the School of Electrical Engineering and Computer Science at Ohio University and the founding director of the VITAL (Virtual Immersive Technologies and Arts for Learning) Lab at Ohio University. Since he joined Ohio University in 2002, he has published over 30 refereed papers and won over 20 grants totalling over 3 million dollars. Sponsors of his projects include the National Science Foundation, Ohio EPA (Environmental Protection Agency), US EPA, Ohio Learning 258
Debra J. Richardson is a professor of Informatics and dean of the Donald Bren School of Information and Computer Sciences, joined the UC Irvine Faculty in 1987 and was appointed chair of ICS in July 2000. Under her leadership, the department was established as the first computingfocused school in the University of California in December 2002. As founding dean of the Donald Bren School of Information and Computer Sciences, she is committed to building the infrastructure to support a dynamic research environment that drives innovation, educates the next generation, contributes to economic competitiveness. She pioneered research in "specification-based testing," whereby formal methods are employed to guide software testing. Her current work focuses on enabling specification-based testing technology throughout the software lifecycle, from requirements and architecture analysis through operation and evolution. She has developed leading edge tools, and has worked with several companies in adopting technology to improve the quality of critical software systems.
Network, ACM (Association for Computing Machinery) SIGCSE (Special Interest Group on Computer Science Education), Filene Research Institute, and Smithsonian Latino Center. He obtained his doctoral degree from the Department of Information & Computer Science at the University of California at Irvine. During his graduate career, he had worked in the Component Applications group at Microsoft Research, the Visual C++ group at Microsoft, and the Personal Security Manager group at AOL/Netscape.
En Ye is a Ph.D. candidate in the School of Electrical Engineering and Computer Science in the Russ College of Engineering and Technology at Ohio University. His research focuses on software model checking (i.e., how to extract software library rules automatically using model checking), and 3-D online virtual worlds (i.e., how to use virtual worlds to enhance collaboration in software development, and software engineering education).
259
