Contracts in Practice

Contracts in Practice? H.-Christian Estler, Carlo A. Furia, Martin Nordio, Marco Piccioni, and Bertrand Meyer Chair of Software Engineering, Department of Computer Science, ETH Zurich, Switzerland

arXiv:1211.4775v6 [cs.SE] 26 Feb 2014

[email protected]

Abstract. Contracts are a form of lightweight formal specification embedded in the program text. Being executable parts of the code, they encourage programmers to devote proper attention to specifications, and help maintain consistency between specification and implementation as the program evolves. The present study investigates how contracts are used in the practice of software development. Based on an extensive empirical analysis of 21 contract-equipped Eiffel, C#, and Java projects totaling more than 260 million lines of code over 7700 revisions, it explores, among other questions: 1) which kinds of contract elements (preconditions, postconditions, class invariants) are used more often; 2) how contracts evolve over time; 3) the relationship between implementation changes and contract changes; and 4) the role of inheritance in the process. It has found, among other results, that: the percentage of program elements that include contracts is above 33% for most projects and tends to be stable over time; there is no strong preference for a certain type of contract element; contracts are quite stable compared to implementations; and inheritance does not significantly affect qualitative trends of contract usage.

1

Introduction

Using specifications as an integral part of the software development process has long been advocated by formal methods pioneers and buffs. While today few people question the value brought by formal specifications, the software projects that systematically deploy them are still a small minority. What can we learn from these adopters about the practical usage of specifications to support software development? In this paper, we answer this question by looking into contracts, a kind of lightweight formal specification in the form of executable assertions (preconditions, postconditions, and class invariants). In the practice of software development, contracts support a range of activities such as runtime checking, automated testing, and static verification, and provide rigorous and unambiguous API documentation. They bring some of the advantages of “heavyweight” formal methods while remaining amenable to programmers without strong mathematical skills: whoever can write Boolean expressions can also write contracts. Therefore, learning how contracts are used in the projects that use them can shed light on how formal methods can make their way into the practice of software development. ?

Work supported by Gebert-Ruf Stiftung, by ERC grant CME # 291389, and by SNF grant ASII # 200021-134976.

The empirical study of this paper analyzes 21 projects written in Eiffel, C#, and Java, three major object-oriented languages supporting contracts, with the goal of studying how formal specifications are written, changed, and maintained as part of general software development. Eiffel has always supported contracts natively; the Java Modeling Language (JML [29]) extends Java with contracts written as comments; and C# has recently added support with the Code Contracts framework [14]. Overall, our study analyzed more than 260 million lines of code and specification distributed over 7700 revisions. To our knowledge, this is the first extensive study of the practical evolving usage of simple specifications such as contracts over project lifetimes. The study’s specific questions target various aspects of how contracts are used in practice: Is the usage of contracts quantitatively significant and uniform across the various selected projects? How does it evolve over time? How does it change with the overall project? What kinds of contracts are used more often? What happens to contracts when implementations change? What is the role of inheritance? The main findings of the study, described in Section 5, include: – The projects in our study make a significant usage of contracts: the percentages of routines and classes with specification is above 33% in the majority of projects. – The usage of specifications tends to be stable over time, except for the occasional turbulent phases where major refactorings are performed. This suggests that contracts evolve following design changes. – There is no strong preference for certain kinds of specification elements (preconditions, postconditions, class invariants); but preconditions, when they are used, tend to be larger (have more clauses) than postconditions. This indicates that different specification elements are used for different purposes. – Specifications are quite stable compared to implementations: a routine’s body may change often, but its contracts will change infrequently. This makes a good case for a fundamental software engineering principle: stable interfaces over changing implementations [41]. – Inheritance does not significantly affect the qualitative findings about specification usage: measures including and excluding inherited contracts tend to correlate. This suggests that the abstraction levels provided by inheritance and by contracts are largely complementary. As a supplemental contribution, we make all data collected for the study available online as an SQL database image [7]. This provides a treasure trove of data about practically all software projects of significant size publicly available that use contracts. Positioning: what this study is not. The term “specification” has a broad meaning. To avoid misunderstandings, let us mention other practices that might be interesting to investigate, but which are not our target in this paper. – We do not consider formal specifications in forms other than executable contracts (Section 2). – We do not look for formal specifications in generic software projects: it is wellknown [42] that the overwhelming majority of software does not come with formal specifications (or any specifications). Instead, we pick our projects among the minority of those actually using contracts, to study how the few adopters use formal specifications in practice. 2

– We do not study applications of contracts; but our analysis may serve as a basis to follow-up studies targeting applications. – We do not compare different methodologies to design and write contracts; we just observe the results of programming practices.

2

Contracts as Specification

1 class MEASURE −− Revision 1 2 3 make do create data end 4 5 data: LIST [INTEGER] −− List of data 6 7 add_datum (d: INTEGER) 8 require not data . has (d) 9 do 10 data . append (d) 11 ensure not data . is_empty 12 13 copy_data ( new_list : LIST [INTEGER]) 14 require new_list 6= Void 15 do 16 across new_list as x: add_datum (x) 17 18 invariant 19 data 6= Void

1 2 3 4 5 6 7 8

class MEASURE −− Revision 2 make do create data end data: LIST [INTEGER] −− List of data add_datum (d: INTEGER) require not data . has (d) d ≥0

9 10 11 12 13 14 15 16 17 18

do data . append (d) ensure not data . is_empty copy_data ( new_list : LIST [INTEGER]) require new_list 6= Void do if not new_list . is_empty then across new_list as x: add_datum (x)

19 20 invariant 21 data 6= Void 22

not data . is_empty

Fig. 1. Class MEASURE in revision 1 (left) and 2 (right). Lines added in revision 2 are shadowed.

Contracts [32] are the form of lightweight specification that we consider in this paper; therefore, we will use the terms “contract” and “specification” as synonyms. This section gives a concise overview of the semantics of contracts and of how they can change with the implementation. The presentation uses a simplified example written in pseudo-Eiffel code (see Figure 1) which is, however, representative of features found in real code (see Section 5). Consider a class MEASURE used to store a sequence of measures, each represented by an integer number; the sequence is stored in a list as attribute data. Figure 1 shows two revisions of class MEASURE, formatted so as to highlight the lines of code or specification added in revision 2. MEASURE includes specification elements in the form of preconditions (require), postconditions (ensure), and class invariants ( invariant ). Each element includes one or more clauses, one per line; the clauses are logically anded. For example, routine (method) add_datum has one precondition clause on line 8 and, in revision 2, another clause on line 9. Contract clauses use the same syntax as Boolean expressions of the programming language; therefore, they are executable and can be checked at runtime. A routine’s 3

precondition must hold whenever the routine is called; the caller is responsible for satisfying the precondition of its callee. A routine’s postcondition must hold whenever the routine terminates execution; the routine body is responsible for satisfying the postcondition upon termination. A class invariant specifies the “stable” object states between consecutive routine calls: it must hold whenever a new object of the class is created and before and after every (public) routine call terminates.1 In Figure 1 (left), routine add_datum must be called with an actual argument d that represents a measure not already stored in the list data (precondition on line 8); when the routine terminates, the list data must not be Void (class invariant on line 19) and not empty (postcondition on line 11). Revision 2 of class MEASURE, in Figure 1 (right), introduces changes in the code and in the specification. Some contracts become stronger: a precondition clause and a class invariant clause are added on lines 9 and 22. Routine copy_data changes its implementation: revision 2 checks whether new_list is empty (line 17) before copying its elements, so as to satisfy the new invariant clause on line 22.

3

Why We Should Care About Changing Specifications

Since specifications in the form of contracts are executable, their changes over the development of a software project may directly affect the ways in which the software is developed, and automatically tested and analyzed. We now sketch a few practical examples that motivate the empirical analysis. Testing is a widely used verification technique based on executing a system to find failures that reveal errors. Testing requires oracles [50,21,19] to determine if the call to a certain routine is valid and produces the expected result. Since contracts can be evaluated at runtime like any other program expression, they can serve as completely automatic testing oracles. Previous work (to mention just a few: [30,6,3,57,52,59,33,55,45]) has built “push button” testing frameworks that use contracts. The effective usage of contracts as oracles in software testing [17,23,1] rests on some assumptions. Besides the obvious requirements that contracts be available and maintained, how pre- and postconditions change affects the testability of routines. The stronger the precondition of a routine r is, the harder is testing the routine, because more calls to r become invalid. In Figure 1, add_datum is harder to test in revision 2 because it has a stronger precondition. On the other hand, a stronger precondition makes r’s clients more easily testable for errors, in that there is a higher chance that a test suite will trigger a precondition violation that does not comply with r’s stricter specification. This is the case of copy_data in the example of Section 2, which calls add_datum and hence may fail to satisfy the latter’s stronger precondition in revision 2. Conversely, a stronger postcondition makes a routine itself easier to test for errors. Conflict analysis. Indirect conflicts [4,38] between code maintained by different developers complicate collaborative development practices. Specifications in the form of contracts can help detect such conflicts: syntactic changes to the contract of a public routine may indicate conflicts in its clients, if the syntactic changes reflect a changed 1

The semantics of class invariants is more subtle in the general case [32] but the details are unimportant here.

4

routine semantics. Thus, using syntactic changes as indicators of possible indirect conflicts is workable only if contracts change much less frequently than implementations, so that following changes in the former generates only a limited number of warnings; and, conversely, only if specifications are consistently changed when the semantics of the implementation changes, so as to produce few false negatives. Object retrieval error detection. Changes in the attributes of a class may affect the capability to retrieve previously stored objects [35,9,43]. Class invariants help detect when inconsistent objects stored in a previous revision are introduced in the system: they express properties of the object state, and hence of attributes, that every valid object must satisfy. In Figure 1, objects stored in revision 1 with an empty data list cannot be retrieved after the code has evolved into revision 2 because they break the new class invariant. Knowing whether developers consistently add invariant clauses for describing constraints on newly introduced attributes tells us whether class invariants are reliable to detect inconsistent objects as they are retrieved.

4

Study Setup

Our study analyzes contract specifications in Eiffel, C#, and Java, covering a wide range of projects of different sizes and life spans developed by professional programmers and researchers. The following subsections present how we selected the projects; the tools we developed to perform the analysis; and the raw measures collected. 4.1

Data Selection

We selected 21 open-source projects that use contracts and are available in public repositories. Save for requiring a minimal amount of revisions (at least 30) and contracts (at least 5% of elements in the latest revisions), we included all open-source projects written in Eiffel, C# with CodeContracts, or Java with JML we could find when we performed this research. Table 2 lists the projects and, for each of them, the total number of REVisions, the life span (AGE, in weeks), the size in lines of code (LOC) at the latest revision, the number of DEVelopers involved (i.e., the number of committers to the repository), and a short description. The 8 Eiffel projects comprise some of the largest publicly available Eiffel applications and libraries, such as the EiffelBase and Gobo libraries (maintained by Eiffel Software and GoboSoft), as well as EiffelProgramAnalysis (developed by PhD students) and AutoTest (developed by our research group). We picked the C# projects available on the Code Contracts webpage [8], which lists all major C# open projects using contracts; 4 of the C# projects (Boogie, CCI, Dafny, and Quickgraph) were mainly developed by Microsoft Research. The main sources of Java projects using JML were the official JML webpage [24], and the OpenJML [39] and KindSoftware [26] projects. We screened 44 projects officially using JML, but only 6 satisfied our requirements about minimal amount of contracts and revisions. With the help of the project configuration files, we manually went through all project repositories to weed out the artifacts not part of the main application (e.g., test suites, accessory library code, or informal documentation). When a repository contained multiple 5

branches, we selected the main branch (trunk in Subversion and default in Mercurial) and excluded the others. # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

PROJECT

LANG .

AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny LabsFramework Quickgraph Rxx Shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Total

Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel C# C# C# C# C# C# C# Java Java Java Java Java Java

# REV. AGE # LOC # DEV. 306 195 65’625 13 1342 1006 61’922 45 208 114 40’750 8 671 747 53’316 8 282 716 21’941 6 120 524 10’840 6 215 716 6’131 7 922 285 163’552 6 766 108 88’284 29 100 171 20’602 3 326 106 29’700 19 49 30 14’540 1 380 100 40’820 4 148 68 55’932 2 59 7 2352 2 376 119 13’294 6 879 366 73’760 27 395 389 35’013 18 29 106 5’963 3 30 350 10’872 7 153 169 15’538 8 7’756 6’392 830’747 228

DESCRIPTION

Contract-based random testing tool General-purpose data structures library Utility library for analyzing Eiffel programs Library for compiler interoperability Portable data structure library Date and time library Library to support design patterns XML Library supporting XSL and XPath Program verification system Library to support compilers construction Program verifier Library to manage experiments in .NET Generic graph data structure library Library of unofficial reactive LINQ extensions Application for messaging in Twitter style Direct Verification Condition Generator An Extended Static Checker for Java (version 2) Front-end parser for Java byte and source code A logging framework Race Condition Checker for Java Editor for Java bytecode and BML specifications

Table 2. List of projects used in the study. “AGE” is in weeks, “# LOC” is lines of code, and “#D EVELOPERS” equals the number of committers to the repository.

4.2

Analysis Tools

To support analysis of large amounts of program code in multiple languages, we developed C OAT—a “COntract Analysis Tool”. The current implementation of C OAT has five main components: C OAT R EPO retrieves the complete revision history of projects; C OAT E IFFEL, C OAT C#, and C OAT JAVA are language-specific back-ends that process Eiffel and C# classes and extract contracts and code into a database; C OATA NALYZE queries the database data supplied by the back-ends and produces the raw measures discussed in Section 4.3. Finally, a set of R scripts read the raw data produced by C OATA NALYZE and perform statistical data analysis. C OAT R EPO accesses Subversion and Mercurial repositories, checks out all revisions of a project, and stores them locally together with other relevant data such as commit dates, messages, and authors. We used this additional data to investigate unexpected behavior, such as sudden extreme changes in project sizes, as we mention in Section 5. C OAT E IFFEL parses Eiffel classes, extracts body and specification elements, and stores them in a relational database, in a form suitable for the subsequent processing. While parsing technology is commonplace, parsing projects over a life span of nearly 20 years (such as EiffelBase) is challenging because of changes in the language syntax and semantics. A major question for our analysis was how to deal with inheritance. Routines and classes inherit contracts as well as implementations; when analyzing the specification of a routine or a class, should our measures include the inherited specification? Since we 6

had no preliminary evidence to prefer one approach or the other, our tools analyze each class twice: once in its flat version and once in its non-flat version. The non-flat version of a class is limited to what appears in the class text. A flat class, in contrast, explicitly includes all the routines (with their specification) and invariants of the ancestor classes. Flattening ignores, however, library classes or framework classes that are not part of the repository. Reconstructing flat classes in the presence of multiple inheritance (supported and frequently used in Eiffel) has to deal with features such as member renaming and redefinitions which introduce further complexity, and hence requires static analysis of the dependencies among the abstract syntax trees of different classes. Parsing C# is simpler because it only deals with the latest version of the language and single inheritance. The Code Contracts library has, however, its peculiarities that require some special processing to make the results comparable with Eiffel’s. For example, specifications of an interface or abstract class must appear in a separate child class containing only the contracts; our tool merges these “specification classes” with their parent. Section 5.6 compares our measures for the flat and non-flat versions of our projects; the overall conclusion is that the measures tend to be correlated. This is a useful piece of information for the continuation of our study: for the measures we took, both considering in detail and overlooking inheritance seem to lead to consistent results. C OATA NALYZE reads the data stored in the database by C OAT E IFFEL, C OAT C#, and C OAT JAVA and computes the raw measures described in Section 4.3. It outputs them to CSV files, which are finally processed by a set of R scripts that produce tables with statistics (such as Table 3) and plots (such as those next to Table 3). The complete set of statistics is available [7,13] (see the appendix). 4.3

Measures

The raw measures produced by C OATA NALYZE include: – the number of classes, the number of classes with invariants, the average number of invariant clauses per class, and the number of classes modified compared to the previous revision; – the number of routines (public and private), the number of routines with nonempty precondition, with non-empty postcondition, and with non-empty specification (that is, precondition, postcondition, or both), the average number of preand postcondition clauses per routine, and the number of routines with modified body compared to the previous revision. Measuring precisely the strength of a specification (which refers to how constraining it is) is hardly possible as it requires detailed knowledge of the semantics of classes and establishing undecidable properties in general (it is tantamount to deciding entailment for a first-order logic theory). In our study, we count the number of specification clauses (elements anded, normally on different lines) as a proxy for specification strength. The number of clauses is a measure of size that is interesting in its own right. Then, if a (non-trivial, i.e., not identically true) clause is added to a specification element without changing its other clauses, we certainly have a strengthening; and, con7

versely, a weakening when we remove a clause. If some clauses are changed,2 just counting the clauses may measure strength incorrectly. We have evidence, however, that the error introduced by measuring strengthening in this way is small. We manually inspected 277 changes randomly chosen, and found 11 misclassifications (e.g., strengthening reported as weakening). Following [31, Eq. 5], this gives a 95% confidence interval of [2%, 7%]: with 95% probability, the errors introduced by our estimate (measuring clauses for strength) involve no more than 7% of the changes.

5

How Contracts Are Used

This section presents the main findings of our study regarding what kinds of specifications programmers write and how they change the specifications as they change the system. Our study targets the following main questions, addressed in each ofthe following subsections. Q1. Do projects make a significant usage of contracts, and how does usage evolve over time? Q2. How does the usage of contracts change with projects growing or shrinking in size? Q3. What kinds of contract elements are used more often? Q4. What is the typical size and strength of contracts, and how does it change over time? Q5. Do implementations change more often than their contracts? Q6. What is the role of inheritance in the way contracts change over time? Table 3 shows the essential quantitative data we discuss for each project; Table 4 shows sample plots of the data for four projects. In the rest of the section, we illustrate and summarize the data in Table 3 and the plots in Table 4 as well as much more data and plots that, for lack of space, are available elsewhere [7,13]. 5.1

Writing Contracts

In the majority of projects in our study, developers devoted a considerable part of their programming effort to writing specifications for their code. While we specifically target projects with some specification (and ignore the majority of software that doesn’t use contracts), we observe that most of the projects achieve significant percentages of routines or classes with specification. As shown in column % ROUTINES SPEC of Table 3, in 7 of the 21 analyzed projects, on average 50% or more of the public routines have some specification (pre- or postcondition); in 14 projects, 35% or more of the routines have specification; and only 3 projects have small percentages of specified routines (15% or less). Usage of class invariants (column % CLASSES INV in Table 3) is more varied but still consistent: in 9 projects, 33% or more of the classes have an invariant; in 10 projects, 12% or less of the classes have an invariant. The standard deviation of these percentages is small for 11 of the 21 projects, compared to the average value over all revisions: the latter is at least five times larger. suggesting that deviations from the 2

We consider all concrete syntactic changes, that is all textual changes.

8

average are normally small. Section 5.2 gives a quantitative confirmation of this hint about the stability of specification amount over time. The EiffelBase project—a large standard library used in most Eiffel projects—is a good “average” example of how contracts may materialize over a project’s lifetime. After an initial fast growing phase (see the first plot in Table 4), corresponding to a still incipient design that is taking shape, the percentages of routines and classes with specification stabilize around the median values with some fluctuations that—while still significant, as we comment on later—do not affect the overall trend or the average percentage of specified elements. This two-phase development (initial mutability followed by stability) is present in several other projects of comparable size, and is sometimes extreme, such as for Boogie, where there is a widely varying initial phase, followed by a very stable one where the percentages of elements with specification is practically constant around 30%. Analyzing the commit logs around the revisions of greater instability showed that wild variations in the specified elements coincide with major reengineering efforts. For Boogie, the initial project phase coincides with the porting of a parent project written in Spec# (a dialect of C#), and includes frequent alternations of adding and removing code from the repository; after this phase, the percentage of routines and classes with specification stabilizes to a value close to the median. There are few outlier projects where the percentage of elements with specification is small, not kept consistent throughout the project’s life, or both. Quickgraph, for example, never has more than 4% of classes with an invariant or routines with a postcondition, and its percentage of routines with precondition varies twice between 12% and 21% in about 100 revisions (see complete data in [13]). In two thirds of the projects, on average 1/3 or more of the routines have some specification (pre- or postconditions). Public vs. private routines. The data analysis focuses on contracts of public routines. To determine whether trends are different for private routines, we visually inspected the plots [7] and computed the correlation coefficient3 τ for the evolution of the percentages of specified public routines against those of private routines. The results suggest to partition the projects into three categories. For the 9 projects in the first category—AutoTest, EiffelBase, Boogie, CCI, Dafny, JavaFE, Logging, RCC and Umbra—the correlation is positive (0.51 ≤ τ ≤ 0.94) and highly significant. The 2 projects in the second category—GoboStructure and Labs—have negative (τ ≤ −0.47) and also significant correlation. The remaining 10 projects belong to the third category, characterized by correlations small in absolute value, positive or negative, or statistically insignificant. This partitioning seems to correspond to different approaches to interface design and encapsulation: for projects in the first category, public and private routines always receive the same amount of specification throughout the project’s life; projects in the second category show negative correlations that may correspond to changes to the visibility status of a significant fraction of the routines; visual inspection of projects in the third category still suggests positive correlations between public and private routines with specification, but the occasional redesign upheaval reduces the overall value of τ or the confidence level. In fact, the confidence level is typically small for projects in the third category; and it is not significant (p = 0.418) only for 3

All correlation measures in the paper employ Kendall’s rank correlation coefficient τ .

9

EiffelProgramAnalysis which also belongs to the third category. Projects with small correlations tend to be smaller in size with fewer routines and classes; conversely, large projects may require a stricter discipline in defining and specifying the interface and its relations with the private parts, and have to adopt consistent approaches throughout their lives. In roughly half of the projects, the amounts of contracts in public and in private routine correlate; in the other half, correlation vanishes due to redesign changes. 5.2

Contracts and Project Size

In Section 5.1, we observed that the percentage of specified routines and classes is fairly stable over time, especially for large projects in their maturity. We analyzed the correlation between measures of elements with specification and project size, and corroborated the found correlations with visual inspection of the graphs. The correlation between the number of routines or classes with some specification and the total number of routines or classes (with or without specification) is consistently strong and highly significant. Looking at routines, 10 projects exhibit an almost perfect correlation with τ > 0.9 and p ∼ 0; only 3 projects show medium/low correlations (Labs and Quickgraph with τ = 0.48, and Logging with τ = 0.32) which are however still significant. The outlook for classes is quite similar: the correlation between number of classes with invariants and number of all classes tends to be high. Outliers are the projects Boogie and JavaFE with the smaller correlations τ = 0.28 and τ = 0.2, but visual inspection still suggests that a sizable correlation exists for Boogie (the results for JavaFE are immaterial since it has only few invariants overall). In all, the absolute number of elements with specification is normally synchronized to the overall size of a project, confirming the suggestion of Section 5.1 that the percentage of routines and classes with specification is stable over time. Having established that, in general, specification and project size have similar trends, we can look into finer-grained variations of specifications over time. To estimate the relative effort of writing specifications, we measured the correlation between percentage of specified routines or classes and number of all routines or all classes. A first large group of projects, almost half of the total whether we look at routines or classes, show weak or negligible correlations (−0.35 < τ < 0.35). In this majority of projects, the relative effort of writing and maintaining specifications evolves largely independently of the project size. Given that the overall trend is towards stable percentages, the high variance often originates from initial stages of the projects when there were few routines or classes in the system and changes can be momentous. GoboKernel and DirectVCGen are specimens of these cases: the percentage of routines with contracts varies wildly in the first 100 revisions when the system is still small and the developers are exploring different design choices and styles. Another group of 3 projects (AutoTest, Boogie, and Dafny) show strong negative correlations (τ < −0.75) both between percentage of specified routines and number of routines and between percentage of specified classes and number of classes. The usual cross-inspection of plots and commit logs points to two independent phenomena that account for the negative correlations. The first is the presence of large merges of project branches into the main branch; these give rise to strong irregularities in the absolute 10

and relative amount of specification used, and may reverse or introduce new specification styles and policies that affect the overall trends. As evident in the second plot of Table 4, AutoTest epitomizes this phenomenon, with its history clearly partitioned into two parts separated by a large merge at revision 150. Before the merge, the system is smaller with high percentages of routines and classes with specification; with the merge, the system grows manifold and continues growing afterward, while the percentage of elements with specification decreases abruptly and then (mostly for class invariants) continues decreasing. The second phenomenon that may account for negative correlations between percentage of specified elements and measures of project size is a sort of “specification fatigue” that kicks in as a project becomes mature and quite large. At that point, there might be diminishing returns for supplying more specification, and so the percentage of elements with specification gracefully decreases while the project grows in size. (This is consistent with Schiller et al.’s suggestion [49] that annotation burden limits the extent to which contracts are used.) The fatigue is, however, of small magnitude if present at all, and may be just be a sign of reached maturity where a solid initial design with plenty of specification elements pays off in the long run to the point that less relative investment is sufficient to maintain a stable level of maintainability and quality. The remaining projects have significant positive correlations (τ > 0.5) between either percentage of specified routines and number of routines or between percentage of specified classes and number of classes, but not both. In these special cases, it looks as if the fraction of programming effort devoted to writing specification tends to increase with the absolute size of the system: when the system grows, proportionally more routines or classes get a specification. However, visual inspection suggests that, in all cases, the trend is ephemeral or contingent on transient phases where the project size changes significantly in little time. As the projects mature and their sizes stabilize, the other two trends (no correlation or negative correlation) emerge in all cases. The fraction of routines and classes with some specification is quite stable over time. Local exceptions are possible when major redesign changes take place. 5.3

Kinds of Contract Elements

Do programmers prefer preconditions? Typically, one would expect that preconditions are simpler to write than postconditions (and, for that matter, class invariants): postconditions are predicates that may involve two states (before and after routine execution). Furthermore, programmers have immediate benefits in writing preconditions as opposed to postconditions: a routine’s precondition defines the valid input; hence, the stronger it is, the fewer cases the routine’s body has to deal with. Contrary to this common assumption, the data in our study (columns % ROUTINES PRE and POST in Table 3) is not consistently lopsided towards preconditions. 2 projects show no difference in the median percentages of routines with precondition and with postcondition. 10 projects do have, on average, more routines with precondition than routines with postcondition, but the difference in percentage is less than 10% in 5 of those projects, and as high as 39% only in one project (Dafny). The remaining 9 projects even have more routines with postcondition than routines with precondition, although the difference is small (less than 5%) in 5 projects, and as high as 45% only in RCC. 11

On the other hand, in 17 projects the percentage of routines with some specification (precondition, postcondition, or both) is higher than both percentages of routines with precondition and of routines with postcondition. Thus, we can partition the routines of most projects in three groups of comparable size: routines with only precondition, routines with only postcondition, and routines with both. The 4 exceptions are CCI, Shweet, DirectVCGen, and Umbra where, however, most elements have little specification. In summary, many exogenous causes may concur to determine the ultimate reasons behind picking one kind of contract element over another, such as the project domain and the different usage of different specification elements. Our data is, however, consistent with the notion that programmers choose which specification to write according to context and requirements, not based on a priori preferences. It is also consistent with Schiller et al.’s observations [49] that contract usage follows different patterns in different projects, and that programmers are reluctant to change their preferred usage patterns—and hence patterns tend to remain consistent within the same project. A closer look at the projects where the difference between percentages of routines with precondition and with postcondition is significant (9% or higher) reveals another interesting pattern. All 6 projects that favor preconditions are written in C# or Java: Dafny, Labs, Quickgraph, Shweet, ESCJava (third plot in Table 4, after rev. 400), and JavaFE; conversely, the 3 of 4 projects that favor postconditions are in Eiffel (AutoTest, GoboKernel, and GoboTime), whereas the fourth is RCC written in Java. A possible explanation for this division involves the longer time that Eiffel has supported contracts and the principal role attributed to Design by Contract within the Eiffel community. C# and Java programmers, then, are more likely to pragmatically go for the immediate tangible benefits brought by preconditions as opposed to postconditions; Eiffel programmers might be more zealous and use contracts thoroughly also for design before implementation. Preconditions and postconditions are used equally frequently across most projects. Class invariants. Class invariants have a somewhat different status than pre- or postconditions. Since class invariants must hold between consecutive routine calls, they define object consistence, and hence they belong to a different category than pre- and postconditions. The percentages of classes with invariant (% CLASSES INV in Table 3) follow similar trends as pre- and postconditions in most projects in our study. Only 4 projects stick out because they have 4% or less of classes with invariant, but otherwise make a significant usage of other specification elements: Quickgraph, EiffelProgramAnalysis, Shweet, and DirectVCGen.4 Compared to the others, Shweet has a short history and EiffelProgramAnalysis involves students as main developers rather than professionals. Given that the semantics of class invariants is less straightforward than that of pre- and postconditions—and can become quite intricate for complex programs [2]— this might be a factor explaining the different status of class invariants in these projects. A specific design style is also likely to influence the usage of class invariants, as we further comment on in Section 5.4. Kinds of constructs. An additional classification of contracts is according to the constructs they use. We gathered data about constructs of three types: expressions in4

While the projects CCI and Umbra have few classes with invariants (4%–6%), we don’t discuss them here because they also only have few routines with preconditions or postconditions.

12

volving checks that a reference is Void (Eiffel) or null (C# and Java); some form of finite quantification (constructs for ∀/∃ over containers exist for all three languages); and old expressions (used in postconditions to refer to values in the pre-state). Void/null checks are by far the most used: in Eiffel, 36%–93% of preconditions, 7%–62% of postconditions, and 14%–86% of class invariants include a Void check; in C#, 80%–96% of preconditions contain null checks, as do 34%–92% of postconditions (the only exception is CCI which does not use postconditions) and 97%–100% of invariants (exceptions are Quickgraph at 20% and Shweet which does not use invariants); in Java, 88%–100% of preconditions, 28%–100% of postconditions, and 50%–77% of class invariants contain null (with the exception of Umbra which has few contracts in general). Void/null checks are simple to write, and hence cost-effective, which explains their wide usage; this may change in the future, with the increasing adoption of static analyses which supersede such checks [34,10]. The predominance of simple contracts and its justification have been confirmed by others [49]. At the other extreme, quantifications are very rarely used: practically never in preor postconditions; and very sparsely (1%–10% of invariants) only in AutoTest, Boogie, Quickgraph, ESCJava, and JavaFE’s class invariants. This may also change in the future, thanks to the progresses in inferring complex contracts [20,54,53], and in methodological support [45]. The usage of old is more varied: C# postconditions practically don’t use it, Java projects rarely use it (2%–3% of postconditions at most), whereas it features in as many as 39% of postconditions for some Eiffel projects. Using old may depend on the design style; for example, if most routines are side-effect free and return a value function solely of the input arguments there is no need to use old. The overwhelming majority of contracts involves Void/null checks. In contrast, quantifiers appear very rarely in contracts. 5.4

Contract Size and Strength

The data about specification size (and strength) partly vindicates the intuition that preconditions are more used. While Section 5.3 showed that routines are not more likely to have preconditions than postconditions, preconditions have more clauses on average than postconditions in all but the 3 projects GoboTime, ESCJava, and Logging. As shown in columns AVG ROUTINES PRE and POST of Table 3, the difference in favor of preconditions is larger than 0.5 clauses in 9 projects, and larger than 1 clause in 3 projects. CCI never deploys postconditions, and hence its difference between preand postcondition clauses is immaterial. GoboTime is a remarkable outlier: not only do twice as many of its routines have a postcondition than have precondition, but its average postcondition has 0.66 more clauses than its average precondition. ESCJava and Logging also have larger postconditions on average but the size difference is less conspicuous (0.25 and 0.32 clauses). We found no simple explanation for these exceptions, but they certainly are the result of deliberate design choices. The following two facts corroborate the idea that programmers tend to do a better job with preconditions than with postconditions—even if they have no general preference for one or another. First, the default “trivial” precondition true is a perfectly 13

reasonable precondition for routines that compute total functions—defined for every value of the input; a trivial postcondition is, in contrast, never satisfactory. Second, in general, “strong” postconditions are more complex than “strong” preconditions [45] since they have to describe more complex relations. Class invariants are not directly comparable to pre- and postconditions, and their usage largely depends on the design style. Class invariants apply to all routines and attributes of a class, and hence they may be used extensively and involve many clauses; conversely, they can also be replaced by pre- and postconditions in most cases, in which case they need not be complex or present at all [40]. In the majority of projects (15 out of 21), however, class invariants have more clauses on average than pre- and postconditions. We might impute this difference to the traditional design principles for objectoriented contract-based programming, which attribute a significant role to class invariants [32,11,46] as the preferred way to define valid object state. In over eighty percent of the projects, the average preconditions contain more clauses than the average postconditions. Section 5.1 observed the prevailing stability over time of routines with specification. Visual inspection and the values of standard deviation point to a qualitatively similar trend for specification size, measured in number of clauses. In the first revisions of a project, it is common to have more varied behavior, corresponding to the system design being defined; but the average strength of specifications typically reaches a plateau, or varies quite slowly, in mature phases. Project Labs is somewhat of an outlier, where the evolution of specification strength over time has a rugged behavior (see [13] for details and plots). Its average number of class invariant clauses has a step at about revision 29, which corresponds to a merge, when it suddenly grows from 1.8 to 2.4 clauses per class. During the few following revisions, however, this figure drops quickly until it reaches a value only slightly higher than what it was before revision 29. What probably happened is that the merge mixed classes developed independently with different programming styles (and, in particular, different attitudes towards the usage of class invariants). Shortly after the merge, the developers refactored the new components to make them comply with the overall style, which is characterized by a certain average invariant strength. One final, qualitative, piece of data about specification strength is that in a few projects there seems to be a moderate increase in the strength of postconditions towards the latest revisions of the project. If this is a real phenomenon, it may show that, as programmers become fluent in writing specification, they are confident enough to go for the more complex postconditions, reversing their initial (moderate) focus on preconditions. This observation is however not applicable to any of the largest and most mature projects we analyzed (e.g., EiffelBase, Boogie, Dafny). The average size (in number of clauses) of specification elements is stable over time.

5.5

Implementation vs. Specification Changes

A phenomenon commonly attributed [42] to specifications is that they are not updated to reflect changes in the implementation: even when programmers deploy specifications 14

extensively in the initial phases of a project, the later become obsolete and, eventually, useless. Is there evidence of such a phenomenon in the data of our projects? We first have to remember a peculiarity of contracts as opposed to other forms of specification. Contracts are executable specifications; normally, they are checked at runtime during debugging and regression testing sessions (and possibly also in production releases, if the overhead is acceptable, to allow for better error reporting from final users). Specifically, most applications and libraries of our study are actively used and maintained. Therefore, their contracts cannot become grossly misaligned with the implementation: inconsistencies quickly generate runtime errors, which can only be fixed by reconciling implementations with their specifications. By and large, the fact that a significant percentage of routines and classes in our study have contracts (Section 5.1) implies that most of them are correct—if incomplete—specifications of routine or class behavior. A natural follow-up question is then whether contracts change more often or less often than the implementations they specify. To answer, we compare two measures in the projects: for each revision, we count the number of routines with changed body and changed specification (pre- or postcondition) and compare it to the number of routines with changed body and unchanged specification. These measures aggregated over all revisions determine a pair of values (cP , uP ) for each project P : cP characterizes the frequency of changes to implementations that also caused a change in the contracts, whereas uP characterizes the frequencies of changes to implementations only. To avoid that few revisions with very many changes dominate the aggregate values for a project, each revision contributes with a binary value to the aggregate value of a project: 0 if no routine has undergone a change of that type in that revision, and 1 otherwise.5 We performed a Wilcoxon signed-rank test comparing the cP ’s to the uP ’s across all projects to determine if the median difference between the two types of events (changed body with and without changed specification) is statistically significant. The results confirm with high statistical significance (V = 0, p = 9.54 · 10−7 , and large effect size—Cohen’s d > 0.99) that specification changes are quite infrequent compared to implementation changes for the same routine. Visual inspection also confirms the same trend: see the last plot in Table 4 about Boogie. A similar analysis ignoring routines with trivial (empty) specification leads to the same conclusion also with statistical significance (V = 29, p = 4.78 · 10−3 , and medium effect size d > 0.5). When specifications do change, what happens to their strength measured in number of clauses? Another Wilcoxon signed-rank test compares the changes to pre- and postconditions and class invariants that added clauses (suggesting strengthening) against those that removed clauses (suggesting weakening). Since changes to specifications are in general infrequent, the results were not as conclusive as those comparing specification and implementation changes. The data consistently points towards strengthening being more frequent than weakening: V = 31.5 and p < 0.02 for precondition changes; V = 29 and p < 0.015 for postcondition changes; V = 58.5 and p = 0.18 for invariant changes. The effect sizes are, however, smallish: Cohen’s d is about 0.4, 0.42, and 0.18 for preconditions, postconditions, and invariants. In all, the effect of strengthening 5

Using other “reasonable” aggregation functions (including exact counting) leads to qualitatively similar results.

15

being more frequent than weakening seems to be real but more data is needed to obtain conclusive evidence. The implementation of an average routine changes much more frequently than its specification. 5.6

Inheritance and Contracts

Inheritance is a principal feature of object-oriented programming, and involves contracts as well as implementations; we now evaluate its effects on the findings previously discussed. We visually inspected the plots and computed correlation coefficients for the percentages and average strength of specified elements in the flat (explicitly including all routines and specification of the ancestor classes) and non-flat (limited to what appears in the class text) versions of the classes. In the overwhelming majority of cases, the correlations are high and statistically significant: 16 projects have τ ≥ 0.54 and p < 10−9 for the percentage of routines with specification; 17 projects have τ ≥ 0.66 and p ∼ 0 for the percentage of classes with invariant; 12 projects have τ ≥ 0.58 and p < 10−7 for the average precondition and postcondition strength (and 7 more projects still have τ ≥ 0.33 and visually evident correlations); and 15 projects have τ ≥ 0.45 and p ∼ 0 for the average invariant strength. The first-order conclusion is that, in most cases, ignoring the inherited specification does not preclude understanding qualitative trends. What about the remaining projects, which have small or insignificant correlations for some of the measures in the flat and non-flat versions? Visual inspection often confirms the absence of significant correlations, in that the measures evolve along manifestly different shapes in the flat or non-flat versions; the divergence in trends is typically apparent in the revisions where the system size changes significantly, where the overall design—and the inheritance hierarchy—is most likely to change. To see if these visible differences invalidate some of the findings discussed so far, we reviewed the findings against the data for flat classes. The big picture was not affected: considering inheritance may affect the measures and offset or bias some trends, but the new measures are still consistent with the same conclusions drawn from the data for nonflat classes. Future work will investigate whether this result is indicative of a mismatch between the semantics of inheritance and how it is used in practice [51,47]. We now discuss the various questions for flat classes in more detail. Qualitative trends of measures involving contracts do not change significantly whether we consider or ignore inherited contracts. Usage (and kinds) of contracts is qualitatively similar for the flat and non-flat classes. Of course, the number of elements with specification tends to be larger in flat classes simply because specifications are inherited. However, the relative magnitude of measures such as the average, minimum, maximum, and standard deviation of routines and classes with specification is quite similar for flat and non-flat. Similar observations hold concerning measures of contract strength and their evolution. Project size is correlated to measures regarding contracts in similar ways in flat and non-flat classes. The few outliers are projects that exhibited a positive correlation between percentage of specified elements and number of elements in the non-flat versions (e.g., τ = 0.66 for EiffelBase); the correlation vanishes in the flat versions (e.g., 16

τ = −0.08 for EiffelBase). As we discussed at the end of Section 5.2, the positive correlations of these outliers were unusual and possibly ephemeral; the fact that correlations dilute away when we consider the inheritance hierarchy reinforces the idea that the positive correlation trends of Section 5.2 are exceptional and largely inconsequential. Java projects are different in that they achieve consistent positive correlations between percentage of specified elements and number of elements in both flat and nonflat versions; there is a simple explanation for this: inheritance hierarchies are shallow in Java projects, and hence inheritance is simply negligible. Change analyses are virtually identical in flat and non-flat classes; this is unsurprising since the analyses (discussed in Section 5.5) target binary differences between a version and the next one, so that the measures gobble up the offset introduced by flattening. Differences between measures with flat and non-flat classes tend to be smaller in C# and Java projects, as opposed to the Eiffel projects. This reveals that multiple inheritance, available in Eiffel but not in C# and Java, may contribute to magnify differences between measures of flat and non-flat classes. (This is also consistent with the observation about shallow inheritance hierarchies in most Java project.)

6

Threats to Validity

Construct validity. The measures taken by C OAT expose two potential threats to construct validity. First, using the number of clauses as a proxy for the strength of a specification may produce imprecise measures; Section 4.3, however, estimated the imprecision and showed it is limited, and hence an acceptable trade-off in most cases (also given that computing strength exactly is infeasible). Besides, the number of clauses is still a valuable size/complexity measure in its own right (Section 5.4). Second, the flattening introduced to study the effect of inheritance (Section 4.2) introduces some approximations when dealing with the most complex usages of multiple inheritance ( select clauses) or of inner classes. We are confident, however, that this approximation has a negligible impact on our measurements as these complex usages occur very rarely. Internal validity. Since we targeted object-oriented languages where inheritance is used pervasively, it is essential that the inheritance structure be taken into account in the measures. We fully addressed this major threat to internal validity by analyzing all projects twice: in non-flat and flat version (Section 5.6). A different threat originates from C OAT failing to parse a few files in some revisions, due to the presence of invalid and outdated language syntax constructs. The impact of this is certainly negligible: less than 0.0069% of all files could not be parsed. Restricting our analysis to the main branches and manually discarding irrelevant content from the repositories pose another potential threat to internal validity. In all cases, we took great care to cover the most prominent development path and to select the main content based on the project configuration files written by the developers, so as to minimize this threat. External validity. Our study is restricted to three formalisms for writing contract specifications: Eiffel, C# with Code Contracts, and Java with JML. While other notations for contracts (e.g., other Java contract libraries or SPARK Ada) are similar, we did 17

not analyze other types of formal specification, which might limit the generalizability of our findings. In contrast, the restriction to open-source projects does not pose a serious threat to external validity in our study, because several of our projects are mainly maintained by professional programmers (EiffelBase and Gobo projects) or by professional researchers in industry (Boogie, CCI, Dafny, and Quickgraph). An important issue to warrant external validity involves the selection of projects. We explicitly targeted projects that make a non-negligible usage of contracts (Section 4.1), as opposed to the overwhelming majority that only include informal documentation or no documentation at all. This deliberate choice limits the generalizability of our findings, but also focuses the study on understanding how contracts can be seriously used in practice. A related observation is that the developers of several of the study’s projects are supporters of using formal specifications. While this is a possible source of bias it also contributes to reliability of the results: since we are analyzing good practices and success stories of writing contracts, we should target competent programmers with sufficient experience, rather than inexpert novices. For the same reason, the most extensive studies on the usage of design patterns (e.g. [48]) target software such as Eclipse— written by experts on design patterns—rather than small student projects or low-quality software. Besides, Schiller et al.’s independent analysis [49] of some C# projects using CodeContracts also included in our study suggests that their developers are hardly fanatic about formal methods, as they use contracts only to the extent that it remains inexpensive and cost-effective, and does not require them to change their programming practices. Nevertheless, to get an idea of whether the programmers we studied really have incomparable skills, we also set up a small control group, consisting of 10 projects developed by students of a software engineering course6 involving students from universities all around the world. The students use Eiffel but usually have limited or no experience with it when starting the course; the selected projects have a development time of circa 6 weeks and numbers of revisions range between 30 to 292. The average size per project is 4185 LOC. All projects were graded good or very good in their final evaluations. We ran the analyses described in this paper on these student projects. In summary (see Section 13 in the Appendix for detailed data), we found that several of the trends measured with the professional programmers were also present in the student projects—albeit on the smaller scale of a course project. This gives some confidence that the big picture outlined by this paper’s results somewhat generalizes to developers willing to spend some programming effort to write contracts and is not limited to “verification wonks” only.

7

Related Work

Section 3 discussed program analysis and other activities where contracts can be useful. To our knowledge, this paper is the first quantitative empirical study of specifications in the form of contracts and their evolution together with code. Schiller et al. [49] study C# projects using CodeContracts (some also part of our study); while our and their results 6

http://se.inf.ethz.ch/research/dose

18

are not directly comparable because we take different measures and classify contract usage differently, the overall qualitative pictures are consistent and nicely complementary. In the paper we also highlighted a few points where their results confirm or justify ours. Schiller et al. do not study contract evolution; there is evidence, however, that other forms of documentation—e.g., comments [15], APIs [25], or tests [58]—evolve with code. A well-known problem is that specification and implementation tend to diverge over time; this is more likely for documents such as requirements and architectural designs that are typically developed and stored separately from the source code. Much research has targeted this problem; specification refinement, for instance, can be applied to software revisions [16]. Along the same lines, some empirical studies analyzed how requirements relate to the corresponding implementations; [22], for example, examines the co-evolution of certain aspects of requirements documents with change logs and shows that topic-based requirements traceability can be automatically implemented from the information stored in version control systems. The information about the usage of formal specification by programmers is largely anecdotal, with the exceptions of a few surveys on industrial practices [5,56]. There is, however, some evidence of the usefulness of contracts and assertions. [28], for example, suggests that increases of assertions density and decreases of fault density correlate. [36] reports that using assertions may decrease the effort necessary for extending existing programs and increase their reliability. In addition, there is evidence that developers are more likely to use contracts in languages that support them natively [5]. As the technology to infer contracts from code reaches high precision levels [12,54], it is natural to compare automatically inferred and programmer-written contracts; they turn out to be, in general, different but with significant overlapping [44]. Our C OAT tool (Section 4.2) is part of a very large family of tools [18] that mine software repositories to extract quantitative data. In particular, it shares some standard technologies with other tools for source code analysis (e.g., [37]).

8

Concluding Discussion & Implications of the Results

Looking at the big picture, our empirical study suggests a few actionable remarks. (i) The effort required to make a quantitatively significant usage of lightweight specifications is sustainable consistently over the lifetime of software projects. This supports the practical applicability of methods and processes that rely on some form of rigorous specification. (ii) The overwhelming majority of contracts that programmers write in practice are short and simple. This means that, to be practical, methods and tools should make the best usage of such simple contracts or acquire more complex and complete specifications by other means (e.g., inference). It also encourages the usage of simple specifications early on in the curriculum and in the training of programmers [27]. (iii) In spite of the simplicity of the contracts that are used in practice, developers who commit to using contracts seem to stick to them over an entire project lifetime. This reveals that even simple specifications bring a value that is worth the effort: a little specification can go a long way. (iv) Developers often seem to adapt their contracts in response to changes in the design; future work in the direction of facilitating these adaptations and making 19

them seamless has a potential for a high impact. (v) A cornerstone software engineering principle—stable interfaces over changing implementations—seems to have been incorporated by programmers. An interesting follow-up question is then whether this principle can be leveraged to improve not only the reusability of software components but also the collaboration between programmers in a development team. (vi) Somewhat surprisingly, inheritance does not seem to affect most qualitative findings of our study. The related important issue of how behavioral subtyping is achieved in practice [47] belongs to future work, together with several other follow-up questions whose answers can build upon the foundations laid by this paper’s results.

9

Conclusions

This paper presented an extensive empirical study of the evolution of specifications in the form of contracts. The study targeted 15 projects written in Eiffel and C# (using Code Contracts) over many years of development. The main results show that the percentages of routines with pre- or postcondition and of classes with invariants is above 33% for most projects; that these percentages tend to be stable over time, if we discount special events like merge or major refactorings; and that specifications change much less often than implementations—which makes a good case for stable interfaces over changing implementations. Acknowledgments. Thanks to Sebastian Nanz for comments on a draft of this paper; and to Todd Schiller, Kellen Donohue, Forrest Coward, and Mike Ernst for sharing a draft of their paper [49] and comments on this work.

References 1. Araujo, W., Briand, L.C., Labiche, Y.: On the effectiveness of contracts as test oracles in the detection and diagnosis of race conditions and deadlocks in concurrent object-oriented software. In: ESEM. pp. 10–19 (2011) 2. Barnett, M., Fähndrich, M., Leino, K.R.M., Müller, P., Schulte, W., Venter, H.: Specification and verification: the Spec# experience. Comm. ACM 54(6), 81–91 (2011) 3. Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on Java predicates. In: ISSTA. pp. 123–133 (2002) 4. Brun, Y., Holmes, R., Ernst, M.D., Notkin, D.: Proactive detection of collaboration conflicts. In: ESEC/FSE. pp. 168–178 (2011) 5. Chalin, P.: Are practitioners writing contracts? In: The RODIN Book. LNCS, vol. 4157, p. 100 (2006) 6. Cheon, Y., Leavens, G.T.: A simple and practical approach to unit testing: The JML and JUnit way. In: ECOOP. pp. 231–255 (2002) 7. http://se.inf.ethz.ch/data/coat/ 8. http://research.microsoft.com/en-us/projects/contracts/ 9. Curino, C., Moon, H.J., Zaniolo, C.: Graceful database schema evolution: the prism workbench. PVLDB 1(1), 761–772 (2008) 10. Dietl, W., Dietzel, S., Ernst, M.D., Muslu, K., Schiller, T.W.: Building and using pluggable type-checkers. In: ICSE. pp. 681–690. ACM (2011)

20

11. Drossopoulou, S., Francalanza, A., Müller, P., Summers, A.J.: A unified framework for verification techniques for object invariants. In: ECOOP 2008 – Object-Oriented Programming. pp. 412–437. Lecture Notes in Computer Science, Springer (2008) 12. Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., Xiao, C.: The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. 69, 35–45 (2007) 13. Estler, H.C., Furia, C.A., Nordio, M., Piccioni, M., Meyer, B.: Contracts in practice. http: //arxiv.org/abs/1211.4775 (2013), extended version with appendix 14. Fähndrich, M., Barnett, M., Logozzo, F.: Embedded contract languages. In: SAC. pp. 2103– 2110. ACM (2010) 15. Fluri, B., Würsch, M., Gall, H.: Do code and comments co-evolve? on the relation between source code and comment changes. In: WCRE. pp. 70–79. IEEE (2007) 16. García-Duque, J., Pazos-Arias, J., López-Nores, M., Blanco-Fernández, Y., Fernández-Vilas, A., Díaz-Redondo, R., Ramos-Cabrer, M., Gil-Solla, A.: Methodologies to evolve formal specifications through refinement and retrenchment in an analysis-revision cycle. Requirements Engineering 14, 129–153 (2009) 17. Gaudel, M.C.: Software testing based on formal specification. In: Testing Techniques in Software Engineering. Lecture Notes in Computer Science, vol. 6153, pp. 215–242. Springer (2007) 18. Germán, D.M., Cubranic, D., Storey, M.A.D.: A framework for describing and understanding mining tools in software development. In: MSR (2005) 19. Harman, M., Kim, S.G., Lakhotia, K., McMinn, P., Yoo, S.: Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem. In: ICST Workshops. pp. 182–191 (2010) 20. Henkel, J., Reichenbach, C., Diwan, A.: Discovering documentation for Java container classes. IEEE Trans. Software Eng. 33(8), 526–543 (2007) 21. Hierons, R.M., Bogdanov, K., Bowen, J.P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P., Lüttgen, G., Simons, A.J.H., Vilkomir, S.A., Woodward, M.R., Zedan, H.: Using formal specifications to support testing. ACM Comput. Surv. 41(2) (2009) 22. Hindle, A., Bird, C., Zimmermann, T., Nagappan, N.: Relating requirements to implementation via topic analysis. In: ICSM (2012) 23. Jiang, Y., Hou, S.S., Shan, J., Zhang, L., Xie, B.: An approach to testing black-box components using contract-based mutation. International Journal of Software Engineering and Knowledge Engineering 18(1), 93–117 (2008) 24. http://www.jmlspecs.org 25. Kim, M., Cai, D., Kim, S.: An empirical investigation into the role of API-level refactorings during software evolution. In: ICSE. pp. 151–160. ACM (2011) 26. http://kindsoftware.com 27. Kiniry, J.R., Zimmerman, D.M.: Secret ninja formal methods. In: FM. LNCS, vol. 5014, pp. 214–228. Springer (2008) 28. Kudrjavets, G., Nagappan, N., Ball, T.: Assessing the relationship between software assertions and faults: An empirical investigation. In: ISSRE. pp. 204–212 (2006) 29. Leavens, G.T., Baker, A.L., Ruby, C.: JML: A notation for detailed design. In: Behavioral Specifications of Businesses and Systems, pp. 175–188. Kluwer Academic Publishers (1999) 30. Marinov, D., Khurshid, S.: TestEra: A novel framework for automated testing of Java programs. In: ASE. p. 22 (2001) 31. Martin, J.K., Hirschberg, D.S.: Small sample statistics for classification error rates II. Tech. rep., CS Department, UC Irvine (1996), http://goo.gl/Ec8oD 32. Meyer, B.: Object Oriented Software Construction. Prentice Hall PTR, 2 edn. (1997)

21

33. Meyer, B., Fiva, A., Ciupa, I., Leitner, A., Wei, Y., Stapf, E.: Programs that test themselves. Computer 42(9), 46–55 (2009) 34. Meyer, B., Kogtenkov, A., Stapf, E.: Avoid a Void: the eradication of null dereferencing. In: Reflections on the Work of C.A.R. Hoare, pp. 189–211. Springer (2010) 35. Monk, S.R., Sommerville, I.: Schema evolution in OODBs using class versioning. SIGMOD Record 22(3), 16–22 (1993) 36. Müller, M.M., Typke, R., Hagner, O.: Two controlled experiments concerning the usefulness of assertions as a means for programming. In: ICSM. pp. 84–92 (2002) 37. Neamtiu, I., Foster, J.S., Hicks, M.W.: Understanding source code evolution using abstract syntax tree matching. In: MSR (2005) 38. Nordio, M., Estler, H.C., Furia, C.A., Meyer, B.: Collaborative software development on the web. http://arxiv.org/abs/1105.0768 (September 2011) 39. http://openjml.org 40. Parkinson, M.: Class invariants: The end of the road? (position paper) (2007) 41. Parnas, D.L.: On the criteria to be used in decomposing systems into modules. Commun. ACM 15(12), 1053–1058 (1972) 42. Parnas, D.L.: Precise documentation: The key to better software. In: The Future of Software Engineering. pp. 125–148. Springer (2011) 43. Piccioni, M., Oriol, M., Meyer, B.: Class schema evolution for persistent object-oriented software: Model, empirical study, and automated support. IEEE TSE (2012), to appear 44. Polikarpova, N., Ciupa, I., Meyer, B.: A comparative study of programmer-written and automatically inferred contracts. In: ISSTA. pp. 93–104 (2009) 45. Polikarpova, N., Furia, C.A., Pei, Y., Wei, Y., Meyer, B.: What good are strong specifications? In: ICSE. pp. 257–266. ACM (2013) 46. Polikarpova, N., Tschannen, J., Furia, C.A., Meyer, B.: Flexible invariants through semantic collaboration. In: FM. LNCS, Springer (2014) 47. Pradel, M., Gross, T.R.: Automatic testing of sequential and concurrent substitutability. In: ICSE. pp. 282–291. ACM (2013) 48. Ricca, F., Penta, M.D., Torchiano, M., Tonella, P., Ceccato, M.: How design notations affect the comprehension of Web applications. Journal of Software Maintenance 19(5), 339–359 (2007) 49. Schiller, T.W., Donohue, K., Coward, F., Ernst, M.D.: Writing and enforcing contract specifications. In: ICSE. ACM (2014) 50. Staats, M., Whalen, M.W., Heimdahl, M.P.E.: Programs, tests, and oracles. In: ICSE. pp. 391–400 (2011) 51. Tempero, E.D., Yang, H.Y., Noble, J.: What programmers do with inheritance in Java. In: ECOOP. Lecture Notes in Computer Science, vol. 7920, pp. 577–601. Springer (2013) 52. Tillmann, N., de Halleux, J.: Pex–White box test generation for .NET. In: TAP. pp. 134–153 (2008) 53. Wasylkowski, A., Zeller, A.: Mining temporal specifications from object usage. Autom. Softw. Eng. 18(3-4), 263–292 (2011) 54. Wei, Y., Furia, C.A., Kazmin, N., Meyer, B.: Inferring better contracts. In: ICSE. pp. 191–200 (2011) 55. Wei, Y., Roth, H., Furia, C.A., Pei, Y., Horton, A., Steindorfer, M., Nordio, M., Meyer, B.: Stateful testing: Finding more errors in code and contracts. In: ASE. IEEE (2011) 56. Woodcock, J., Larsen, P.G., Bicarregui, J., Fitzgerald, J.: Formal methods: Practice and experience. ACM CSUR 41(4) (2009) 57. Xie, T.: Augmenting automatically generated unit-test suites with regression oracle checking. In: ECOOP. Lecture Notes in Computer Science, vol. 4067, pp. 380–403. Springer (2006) 58. Zaidman, A., Van Rompaey, B., Demeyer, S., van Deursen, A.: Mining software repositories to study co-evolution of production and test code. In: ICST. pp. 220 –229 (2008)

22

59. Zimmerman, D.M., Nagmoti, R.: JMLUnit: The next generation. In: FoVeOOS. LNCS, vol. 6528, pp. 183–197. Springer (2010)

23

# CLASSES m µ M σ 98 220 254 66 93 184 256 36 0 179 221 30 0 72 157 38 42 75 109 17 0 22 47 10 3 25 43 10 0 176 859 252 9 606 647 181 45 60 108 15 11 148 184 25 47 58 75 8 228 260 336 27 0 145 189 53 0 28 36 13 13 55 82 17 66 161 308 80 107 124 641 29 20 22 23 1 48 142 144 42 23 41 77 16

% CLASSES INV m µ M σ 0.38 0.43 0.55 0.06 0.24 0.34 0.39 0.03 0 0.04 0.05 0 0 0.11 0.13 0.04 0.19 0.33 0.39 0.06 0 0.12 0.28 0.09 0 0.22 0.5 0.08 0 0.38 0.48 0.07 0.24 0.34 0.58 0.06 0.01 0.04 0.06 0.01 0.04 0.47 0.52 0.06 0.35 0.38 0.42 0.02 0 0.02 0.04 0.01 0 0.42 0.44 0.08 0 0 0 0 0 0 0.03 0 0.11 0.17 0.26 0.05 0.12 0.47 0.62 0.04 0.04 0.09 0.09 0.01 0.08 0.1 0.11 0.01 0 0.06 0.1 0.03 m 352 545 0 0 122 0 1 0 80 160 25 351 1074 0 0 74 233 499 154 359 36

# ROUTINES µ M σ 1053 1234 372 1984 3323 696 828 1127 199 168 702 155 372 483 88 176 333 53 90 185 55 883 5465 1603 3542 3748 1055 210 302 50 375 551 85 413 518 29 1262 1862 179 1358 1792 494 57 85 33 440 582 115 585 3079 853 589 1081 125 171 173 6 441 447 35 122 332 78

% ROUTINES SPEC m µ M σ 0.47 0.49 0.61 0.06 0.26 0.4 0.44 0.04 0 0.25 0.27 0.02 0 0.6 1 0.17 0.18 0.29 0.41 0.07 0 0.63 0.66 0.06 0 0.9 0.98 0.14 0 0.35 0.44 0.05 0.49 0.52 0.81 0.09 0 0.03 0.05 0.01 0.16 0.64 0.74 0.07 0.38 0.47 0.5 0.03 0 0.16 0.22 0.07 0 0.7 0.97 0.11 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.16 0.36 0.74 0.21 0.34 0.43 0.8 0.15 0.32 0.49 0.54 0.04 0.06 0.56 0.59 0.24 0 0.02 0.05 0.02

% ROUTINES PRE m µ M σ 0.23 0.25 0.4 0.07 0.17 0.27 0.3 0.03 0 0.14 0.16 0.02 0 0.3 0.4 0.09 0.07 0.19 0.28 0.06 0 0.28 0.33 0.03 0 0.58 0.83 0.12 0 0.23 0.35 0.03 0.28 0.3 0.74 0.13 0 0.03 0.04 0.01 0.16 0.57 0.64 0.06 0.28 0.38 0.42 0.03 0 0.15 0.21 0.07 0 0.6 0.93 0.13 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.14 0.27 0.69 0.2 0.26 0.34 0.74 0.14 0.14 0.33 0.35 0.04 0.03 0.07 0.1 0.02 0 0.01 0.03 0.01

% ROUTINES POST m µ M σ 0.34 0.36 0.45 0.04 0.14 0.24 0.26 0.03 0 0.15 0.16 0.01 0 0.51 1 0.19 0.16 0.23 0.32 0.05 0 0.58 0.6 0.06 0 0.58 0.67 0.11 0 0.23 0.33 0.06 0.08 0.32 0.38 0.04 0 0 0.01 0 0 0.18 0.22 0.03 0.1 0.13 0.21 0.03 0 0.01 0.02 0.01 0 0.62 0.81 0.08 0 0.01 0.07 0.02 0.02 0.1 0.35 0.05 0.06 0.12 0.2 0.03 0.13 0.18 0.31 0.04 0.21 0.28 0.33 0.02 0.04 0.52 0.54 0.23 0 0.02 0.04 0.01

200

400

800

revision

600

1000

1200 inv

pre post0 50

100 revision

150

200

250

300

AutoTest: % routines and classes with specification

# clauses per routine or class

% routines or classes

0.40

m 1.73 1.43 0 0 1.45 0 0 0 1.6 1 1 1.34 0 0 0 1 1.07 1.2 1.39 1.21 0

µ 1.76 1.6 1.23 2.1 1.82 1.62 1.8 1.43 1.73 1.33 2.29 1.37 1.71 2.1 1.6 1 1.27 1.54 1.43 1.28 1

M 1.85 1.7 1.25 2.91 1.93 1.7 2.07 1.55 1.76 1.6 2.36 1.58 2.1 2.24 2 1.33 1.66 1.61 1.5 1.36 1

σ 0.03 0.05 0.09 0.59 0.13 0.15 0.24 0.14 0.03 0.22 0.18 0.08 0.71 0.18 0.77 0.05 0.21 0.12 0.04 0.04 0.49

m 1.19 1.2 0 0 1.17 0 0 0 1 0 0 1.13 0 0 0 1 1.21 1.26 1.58 1 0

µ 1.22 1.46 1.13 1.32 1.44 2.28 1.29 1.2 1.02 0 1.04 1.17 1.18 1.03 1 1 1.52 1.48 1.75 1.04 1

M 1.28 1.51 1.17 1.86 1.49 2.53 1.52 1.36 1.02 1 1.05 1.28 1.36 1.12 1 1 1.88 1.82 2 1.05 1

inv

pre 0 post

200

revision

400

600

800

ESCJava: Specification clauses per routine or class

spec only

body+spec body only200

400 revision

600

800

Boogie: # changed routines or spec

σ 0.03 0.06 0.08 0.25 0.1 0.25 0.25 0.07 0.01 0.49 0.14 0.05 0.46 0.1 0.49 0 0.12 0.09 0.08 0.02 0.47

AVG ROUTINES PRE AVG ROUTINES POST

Table 4. Selected plots for projects EiffelBase, AutoTest, ESCJava, and Boogie. Each graph from left to right represents the evolution over successive revisions of: (1) and (2), percentage of routines with precondition (pre in the legend), with postcondition (post), and of classes with invariant (inv); (3), average number of clauses in contracts; (4), number of changes to implementation and specification (body+spec), to implementation only (body only), and change to specification only. When present, a thin gray line plots the total number of routine in the project (scaled). Similar plots for all projects are available [13,7].

inv

pre post0

EiffelBase: % routines and classes with specification

# routines

Table 3. Specification overall statistics with non-flat classes. For each project, we report the number of classes and of public routines (# CLASSES, # ROUTINES); the percentage (1 is 100%) of classes with non-empty invariant (% CLASSES INV); of routines with non-empty specification (% ROU TINES SPEC ) and more specifically with non-empty precondition ( PRE ) and postcondition ( POST); the mean number of clauses of routine preconditions (AVG ROUTINES PRE) and of postconditions (POST). For each measure, the table reports minimum (m), median (µ), maximum (M), and standard deviation (σ) across all revisions.

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx Shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra

% routines or classes

0.35

0.30

0.25

0.20

0.15

0.55 0.50 0.45 0.40 0.35 0.30 0.25

0.8 0.6 0.4 0.2

150 100 50 0

24

10

Appendix: Complete Statistics

This appendix contains several tables with all statistics discussed in the paper. In all tables, the few missing data about project Shweet are due to the fact that the project lacks class invariants, and hence the corresponding statistics are immaterial. General specification statistics. Table 95 lists various general statistics about specifications: # of classes, % of classes with invariant, # of routines, % of routines with specification (pre- or postcondition), % of routines with precondition, % of routines with postcondition, average number of clauses in preconditions, average number of clauses in postconditions, average number of clauses in class invariants. Table 96 lists the same data but for flat classes. Table 97 lists other statistics about projects: language, number of revisions, age in weeks, lines of code, and then some of the same statistics about classes and routines as in Table 95. Table 98 lists the same data for flat classes. Change correlation analysis. Table 99 lists Wilcoxon signed-rank tests about changes, as described in Section 5.5, comparing: changing and non-changing specifications of routines whose body changes; changing and non-changing NE (i.e., non-empty) specifications of routines whose body changes; preconditions becoming weaker vs. becoming stronger, NE preconditions becoming weaker vs. becoming stronger, postconditions becoming weaker vs. becoming stronger, NE postconditions becoming weaker vs. becoming stronger, class invariants becoming weaker vs. becoming stronger, NE class invariants becoming weaker vs. becoming stronger, class invariants becoming weaker when attributes are added vs. it not changing in strength; class invariants becoming stronger when attributes are added vs. it not changing in strength; class invariants becoming weaker when attributes are removed vs. it not changing in strength; class invariants becoming stronger when attributes are removed vs. it not changing in strength. The statistics are V and p from the signed-rank test; ∆(µ) is the difference in medians, whose value is positive iff the first—between the two compared measures—median is larger; d is Cohen’s effect size ((m1 − m2 )/σ where m1 , m2 are the means of the two compared measures and σ is the standard deviation of the whole measured data), whose value is positive iff the first—between the two compared measures—mean is larger. The top half of the table considers non-flat classes, whereas the bottom half considers flat classes. Table 100 shows the results of the same analysis but done by summing all changes instead of counting them with a binary value for each revision (see Section 5.5). Flat vs. non-flat correlation analysis. Table 101 lists correlation statistics between the evolution of the following measures in flat and non-flat classes: % of routines with specification; % of classes with invariants; average number of precondition clauses; average number of postcondition clauses, average number of invariant clauses. Size correlation analysis. Table 102 lists correlation statistics between several pair of measures: % of routines with specification and total # of routines; # of routines with specification and total # of routines; % of classes with invariant and total # of classes; # of classes with invariant and total # of classes, average number of precondition clauses and % of routines with precondition; average number of postcondition clauses and % of routines with postcondition; average number of invariant clauses and % of classes with invariants. Table 103 lists the same statistics for flat classes. 25

Public vs. non-public correlation analysis. Table 104 lists correlation statistics between: % of public routines with precondition and % of non-public routines with precondition; % of public routines with postcondition and % of non-public routines with postcondition; % of public routines with specification and % of non-public routines with specification. Table 105 lists the same statistics for flat classes. Kinds of constructs. Table 106 lists statistics about constructs used in contracts: % of preconditions with Void or null checks; % of preconditions with some form of quantification; % of postconditions with Void or null checks; % of postconditions with some form of quantification; % of postconditions with old; % of class invariants with Void or null checks; % of class invariants with some form of quantification.

26

# CLASSES % CLASSES INV # ROUTINES Project m µ M σ m µ M σ m µ M AutoTest 98 220 254 66 0.38 0.43 0.55 0.06 352 1053 1234 EiffelBase 93 184 256 36 0.24 0.34 0.39 0.03 545 1984 3323 EiffelProgramAnalysis 0 179 221 30 0 0.04 0.05 0 0 828 1127 GoboKernel 0 72 157 38 0 0.11 0.13 0.04 0 168 702 GoboStructure 42 75 109 17 0.19 0.33 0.39 0.06 122 372 483 GoboTime 0 22 47 10 0 0.12 0.28 0.09 0 176 333 GoboUtility 3 25 43 10 0 0.22 0.5 0.08 1 90 185 GoboXML 0 176 859 252 0 0.38 0.48 0.07 0 883 5465 Boogie 9 606 647 181 0.24 0.34 0.58 0.06 80 3542 3748 CCI 45 60 108 15 0.01 0.04 0.06 0.01 160 210 302 Dafny 11 148 184 25 0.04 0.47 0.52 0.06 25 375 551 Labs 47 58 75 8 0.35 0.38 0.42 0.02 351 413 518 Quickgraph 228 260 336 27 0 0.02 0.04 0.01 1074 1262 1862 Rxx 0 145 189 53 0 0.42 0.44 0.08 0 1358 1792 shweet 0 28 36 13 0 0 0 0 0 57 85 DirectVCGen 13 55 82 17 0 0 0.03 0 74 440 582 ESCJava 66 161 308 80 0.11 0.17 0.26 0.05 233 585 3079 JavaFE 107 124 641 29 0.12 0.47 0.62 0.04 499 589 1081 Logging 20 22 23 1 0.04 0.09 0.09 0.01 154 171 173 RCC 48 142 144 42 0.08 0.1 0.11 0.01 359 441 447 Umbra 23 41 77 16 0 0.06 0.1 0.03 36 122 332 σ 372 696 199 155 88 53 55 1603 1055 50 85 29 179 494 33 115 853 125 6 35 78

% ROUTINES SPEC m µ M σ 0.47 0.49 0.61 0.06 0.26 0.4 0.44 0.04 0 0.25 0.27 0.02 0 0.6 1 0.17 0.18 0.29 0.41 0.07 0 0.63 0.66 0.06 0 0.9 0.98 0.14 0 0.35 0.44 0.05 0.49 0.52 0.81 0.09 0 0.03 0.05 0.01 0.16 0.64 0.74 0.07 0.38 0.47 0.5 0.03 0 0.16 0.22 0.07 0 0.7 0.97 0.11 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.16 0.36 0.74 0.21 0.34 0.43 0.8 0.15 0.32 0.49 0.54 0.04 0.06 0.56 0.59 0.24 0 0.02 0.05 0.02

% ROUTINES PRE m µ M σ 0.23 0.25 0.4 0.07 0.17 0.27 0.3 0.03 0 0.14 0.16 0.02 0 0.3 0.4 0.09 0.07 0.19 0.28 0.06 0 0.28 0.33 0.03 0 0.58 0.83 0.12 0 0.23 0.35 0.03 0.28 0.3 0.74 0.13 0 0.03 0.04 0.01 0.16 0.57 0.64 0.06 0.28 0.38 0.42 0.03 0 0.15 0.21 0.07 0 0.6 0.93 0.13 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.14 0.27 0.69 0.2 0.26 0.34 0.74 0.14 0.14 0.33 0.35 0.04 0.03 0.07 0.1 0.02 0 0.01 0.03 0.01

% ROUTINES POST m µ M σ 0.34 0.36 0.45 0.04 0.14 0.24 0.26 0.03 0 0.15 0.16 0.01 0 0.51 1 0.19 0.16 0.23 0.32 0.05 0 0.58 0.6 0.06 0 0.58 0.67 0.11 0 0.23 0.33 0.06 0.08 0.32 0.38 0.04 0 0 0.01 0 0 0.18 0.22 0.03 0.1 0.13 0.21 0.03 0 0.01 0.02 0.01 0 0.62 0.81 0.08 0 0.01 0.07 0.02 0.02 0.1 0.35 0.05 0.06 0.12 0.2 0.03 0.13 0.18 0.31 0.04 0.21 0.28 0.33 0.02 0.04 0.52 0.54 0.23 0 0.02 0.04 0.01

µ 1.76 1.6 1.23 2.1 1.82 1.62 1.8 1.43 1.73 1.33 2.29 1.37 1.71 2.1 1.6 1 1.27 1.54 1.43 1.28 1

M 1.85 1.7 1.25 2.91 1.93 1.7 2.07 1.55 1.76 1.6 2.36 1.58 2.1 2.24 2 1.33 1.66 1.61 1.5 1.36 1

σ 0.03 0.05 0.09 0.59 0.13 0.15 0.24 0.14 0.03 0.22 0.18 0.08 0.71 0.18 0.77 0.05 0.21 0.12 0.04 0.04 0.49

AVG ROUTINES PRE

m 1.73 1.43 0 0 1.45 0 0 0 1.6 1 1 1.34 0 0 0 1 1.07 1.2 1.39 1.21 0

Table 5. Specification overall statistics with non-flattened classes.

µ 1.22 1.46 1.13 1.32 1.44 2.28 1.29 1.2 1.02 0 1.04 1.17 1.18 1.03 1 1 1.52 1.48 1.75 1.04 1

M 1.28 1.51 1.17 1.86 1.49 2.53 1.52 1.36 1.02 1 1.05 1.28 1.36 1.12 1 1 1.88 1.82 2 1.05 1

σ 0.03 0.06 0.08 0.25 0.1 0.25 0.25 0.07 0.01 0.49 0.14 0.05 0.46 0.1 0.49 0 0.12 0.09 0.08 0.02 0.47

AVG ROUTINES POST

m 1.19 1.2 0 0 1.17 0 0 0 1 0 0 1.13 0 0 0 1 1.21 1.26 1.58 1 0

m 2.76 2.08 0 0 1.78 0 0 0 2.6 1 1 1.76 0 0 0 0 2.59 3.87 1 4.5 0

M 3.51 2.62 1.33 2 3.54 3.27 5.8 2.49 5.32 2 2.94 2.38 2.14 9 0 2 3.56 5.03 2 6 2.25

AVG INV

µ 3.07 2.3 1.17 1.44 2.88 2.5 3.82 2.1 3.14 1 2.8 1.9 2 3.16 0 0 2.77 4.37 1 4.5 1.75

σ 0.22 0.13 0.12 0.41 0.48 0.41 1.12 0.38 0.45 0.35 0.25 0.15 0.75 1.12 0 0.1 0.25 0.26 0.26 0.54 0.82

27

28

µ 1.13 1.49 1.13 1.43 1.53 2.21 1.19 1.13 1.02 0 1.01 1.12 1.18 1.03 1 1 1.44 1.44 1.75 1.02 1

M 1.21 1.53 1.59 1.66 1.61 2.51 1.37 1.3 1.03 1 1.02 1.19 1.36 1.12 1 1 1.74 4.59 2 1.03 1

σ 0.04 0.04 0.17 0.24 0.04 0.26 0.2 0.07 0.01 0.49 0.14 0.03 0.46 0.09 0.49 0 0.11 0.41 0.08 0.01 0.47

AVG ROUTINES POST

σ 0.03 0.03 0.11 0.65 0.13 0.18 0.19 0.12 0.05 0.22 0.16 0.08 0.71 0.18 0.77 0.05 0.19 0.06 0.04 0.04 0.49

m 1.11 1.27 0 0 1.46 0 0 0 1 0 0 1.09 0 0 0 1 1.15 1.32 1.58 1 0

M 1.82 1.45 1.34 3.31 1.95 1.7 2.07 1.54 1.76 1.6 2.21 1.44 2.1 2.24 2 1.33 1.61 1.48 1.5 1.37 1

m 1.7 1.3 0 0 1.42 0 0 0 1.49 1 1 1.17 0 0 0 1 1.06 1.23 1.39 1.23 0

µ 1.74 1.38 1.16 2.26 1.61 1.61 1.7 1.34 1.51 1.33 2.09 1.19 1.71 2.01 1.6 1 1.23 1.42 1.43 1.31 1

AVG ROUTINES PRE

Table 6. Specification overall statistics with flattened classes.

# CLASSES % CLASSES INV # ROUTINES % ROUTINES SPEC % ROUTINES PRE % ROUTINES POST Project m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ AutoTest 98 220 254 66 0.48 0.54 0.63 0.05 524 2395 2946 1005 0.58 0.6 0.77 0.06 0.37 0.39 0.48 0.04 0.38 0.41 0.59 0.06 EiffelBase 93 184 256 36 0.49 0.66 0.69 0.04 1377 5214 7542 1326 0.54 0.68 0.69 0.03 0.38 0.48 0.49 0.02 0.3 0.45 0.49 0.03 EiffelProgramAnalysis 0 179 221 30 0 0.07 0.08 0.01 0 5300 8360 2064 0 0.2 0.25 0.03 0 0.11 0.18 0.03 0 0.1 0.13 0.02 GoboKernel 0 72 157 38 0 0.12 0.29 0.04 0 271 1304 308 0 0.84 1 0.15 0 0.49 0.6 0.16 0 0.62 1 0.18 GoboStructure 42 75 109 17 0.5 0.81 0.87 0.11 245 1421 2900 696 0.7 0.83 0.91 0.04 0.51 0.57 0.74 0.05 0.53 0.61 0.76 0.06 GoboTime 0 22 47 10 0 0.18 0.38 0.1 0 346 630 87 0 0.85 0.88 0.08 0 0.46 0.53 0.05 0 0.69 0.73 0.07 GoboUtility 3 25 43 10 0 0.52 0.57 0.09 1 118 221 62 0 0.91 0.99 0.12 0 0.5 0.83 0.09 0 0.68 0.72 0.12 GoboXML 0 176 859 252 0 0.63 0.8 0.13 0 2326 113813 35183 0 0.67 0.77 0.09 0 0.49 0.67 0.08 0 0.4 0.52 0.11 Boogie 9 606 647 181 0.24 0.34 0.58 0.06 80 7032 7894 2249 0.63 0.65 0.8 0.05 0.5 0.5 0.74 0.06 0.08 0.43 0.48 0.06 CCI 45 60 108 15 0.01 0.04 0.06 0.01 163 214 309 52 0 0.03 0.05 0.01 0 0.03 0.04 0.01 0 0 0.01 0 Dafny 11 148 184 25 0.04 0.47 0.52 0.06 25 886 1515 309 0.16 0.62 0.72 0.09 0.16 0.46 0.53 0.06 0 0.27 0.35 0.06 Labs 47 58 75 8 0.35 0.38 0.42 0.02 684 790 941 51 0.31 0.49 0.53 0.06 0.19 0.37 0.43 0.06 0.12 0.16 0.19 0.02 Quickgraph 228 260 336 27 0 0.02 0.04 0.01 1075 1263 1863 179 0 0.16 0.22 0.07 0 0.15 0.21 0.07 0 0.01 0.02 0.01 Rxx 0 145 189 53 0 0.42 0.44 0.08 0 1448 2686 777 0 0.77 0.98 0.1 0 0.69 0.95 0.11 0 0.72 0.86 0.08 shweet 0 28 36 13 0 0 0 0 0 67 100 39 0 0.09 0.4 0.07 0 0.08 0.4 0.07 0 0.01 0.07 0.01 DirectVCGen 13 55 82 17 0 0 0.03 0 274 680 1199 200 0.02 0.11 0.18 0.02 0.02 0.11 0.18 0.02 0.01 0.08 0.18 0.03 ESCJava 66 161 308 80 0.11 0.17 0.26 0.05 304 746 3819 1130 0.17 0.42 0.76 0.21 0.13 0.25 0.68 0.2 0.08 0.14 0.35 0.04 JavaFE 107 124 641 29 0.12 0.47 0.62 0.04 815 980 1761 176 0.37 0.45 0.74 0.12 0.3 0.38 0.67 0.11 0.18 0.22 0.31 0.03 Logging 20 22 23 1 0.04 0.09 0.09 0.01 154 171 173 6 0.32 0.49 0.54 0.04 0.14 0.33 0.35 0.04 0.21 0.28 0.33 0.02 RCC 48 142 144 42 0.08 0.1 0.11 0.01 798 952 959 66 0.04 0.69 0.7 0.31 0.02 0.04 0.05 0.01 0.04 0.67 0.67 0.3 Umbra 23 41 77 16 0 0.06 0.1 0.03 335 612 1466 355 0 0.03 0.06 0.02 0 0.02 0.05 0.02 0 0.01 0.02 0.01

m 3.42 6.75 0 0 3.67 0 0 0 2.6 1 1 2.63 0 0 0 0 3.04 4.55 1 4.5 0

AVG INV

µ M σ 3.58 3.81 0.14 10.8 13.27 1.16 1.1 1.21 0.1 1.4 2 0.41 6.74 8.61 1.32 2.33 5.13 1.17 3 3.36 0.58 2.92 8.96 2.49 3.79 5.66 0.44 1 2 0.35 3.71 4.14 0.41 3.14 3.4 0.19 2 2.14 0.75 3.16 9 1.12 0 0 0 0 2 0.1 3.58 5.57 0.26 4.96 6 0.39 1 2 0.26 4.5 6 0.54 3.75 6.25 2.22

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra

lang Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel C# C# C# C# C# C# C# Java Java Java Java Java Java

rev 306 1342 208 671 282 120 215 922 766 100 326 49 380 148 59 376 879 395 29 30 153

age 195 1006 114 747 716 524 716 285 108 171 106 30 100 68 7 119 366 389 106 350 169

m 17154 13154 0 0 3995 0 188 0 1831 6352 847 10296 22421 32 37 1113 15070 22813 4965 9362 2136

LOC µ M 52958 65625 45200 61922 27498 40750 8457 53316 13069 21941 4131 10840 3591 6131 24146 163552 83444 88284 8358 20602 21700 29700 12141 14540 30343 40820 41616 55932 1741 2352 9550 13294 42055 73760 24993 35013 5751 5963 10688 10872 7549 15538 σ 19291 11922 8546 9175 4525 1753 1816 48763 23486 2834 4709 1002 4767 15998 887 3645 17731 2634 254 531 3723

# CLASSES m µ M σ 98 220 254 66 93 184 256 36 0 179 221 30 0 72 157 38 42 75 109 17 0 22 47 10 3 25 43 10 0 176 859 252 9 606 647 181 45 60 108 15 11 148 184 25 47 58 75 8 228 260 336 27 0 145 189 53 0 28 36 13 13 55 82 17 66 161 308 80 107 124 641 29 20 22 23 1 48 142 144 42 23 41 77 16

% CLASSES INV m µ M σ 0.38 0.43 0.55 0.06 0.24 0.34 0.39 0.03 0 0.04 0.05 0 0 0.11 0.13 0.04 0.19 0.33 0.39 0.06 0 0.12 0.28 0.09 0 0.22 0.5 0.08 0 0.38 0.48 0.07 0.24 0.34 0.58 0.06 0.01 0.04 0.06 0.01 0.04 0.47 0.52 0.06 0.35 0.38 0.42 0.02 0 0.02 0.04 0.01 0 0.42 0.44 0.08 0 0 0 0 0 0 0.03 0 0.11 0.17 0.26 0.05 0.12 0.47 0.62 0.04 0.04 0.09 0.09 0.01 0.08 0.1 0.11 0.01 0 0.06 0.1 0.03

m 352 545 0 0 122 0 1 0 80 160 25 351 1074 0 0 74 233 499 154 359 36

# ROUTINES µ M σ 1053 1234 372 1984 3323 696 828 1127 199 168 702 155 372 483 88 176 333 53 90 185 55 883 5465 1603 3542 3748 1055 210 302 50 375 551 85 413 518 29 1262 1862 179 1358 1792 494 57 85 33 440 582 115 585 3079 853 589 1081 125 171 173 6 441 447 35 122 332 78

% ROUTINES SPEC m µ M σ 0.47 0.49 0.61 0.06 0.26 0.4 0.44 0.04 0 0.25 0.27 0.02 0 0.6 1 0.17 0.18 0.29 0.41 0.07 0 0.63 0.66 0.06 0 0.9 0.98 0.14 0 0.35 0.44 0.05 0.49 0.52 0.81 0.09 0 0.03 0.05 0.01 0.16 0.64 0.74 0.07 0.38 0.47 0.5 0.03 0 0.16 0.22 0.07 0 0.7 0.97 0.11 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.16 0.36 0.74 0.21 0.34 0.43 0.8 0.15 0.32 0.49 0.54 0.04 0.06 0.56 0.59 0.24 0 0.02 0.05 0.02

Table 7. Project specification statistics for non-flattened classes.

% ROUTINES PRE m µ M σ 0.23 0.25 0.4 0.07 0.17 0.27 0.3 0.03 0 0.14 0.16 0.02 0 0.3 0.4 0.09 0.07 0.19 0.28 0.06 0 0.28 0.33 0.03 0 0.58 0.83 0.12 0 0.23 0.35 0.03 0.28 0.3 0.74 0.13 0 0.03 0.04 0.01 0.16 0.57 0.64 0.06 0.28 0.38 0.42 0.03 0 0.15 0.21 0.07 0 0.6 0.93 0.13 0 0.1 0.4 0.07 0.06 0.15 0.37 0.04 0.14 0.27 0.69 0.2 0.26 0.34 0.74 0.14 0.14 0.33 0.35 0.04 0.03 0.07 0.1 0.02 0 0.01 0.03 0.01

% ROUTINES POST m µ M σ 0.34 0.36 0.45 0.04 0.14 0.24 0.26 0.03 0 0.15 0.16 0.01 0 0.51 1 0.19 0.16 0.23 0.32 0.05 0 0.58 0.6 0.06 0 0.58 0.67 0.11 0 0.23 0.33 0.06 0.08 0.32 0.38 0.04 0 0 0.01 0 0 0.18 0.22 0.03 0.1 0.13 0.21 0.03 0 0.01 0.02 0.01 0 0.62 0.81 0.08 0 0.01 0.07 0.02 0.02 0.1 0.35 0.05 0.06 0.12 0.2 0.03 0.13 0.18 0.31 0.04 0.21 0.28 0.33 0.02 0.04 0.52 0.54 0.23 0 0.02 0.04 0.01

29

30

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra

lang Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel C# C# C# C# C# C# C# Java Java Java Java Java Java

rev 306 1342 208 671 282 120 215 922 766 100 326 49 380 148 59 376 879 395 29 30 153

age 195 1006 114 747 716 524 716 285 108 171 106 30 100 68 7 119 366 389 106 350 169

m 17154 13154 0 0 3995 0 188 0 1831 6352 847 10296 22421 32 37 1113 15070 22813 4965 9362 2136

σ 19291 11922 8546 9175 4525 1753 1816 48763 23486 2834 4709 1002 4767 15998 887 3645 17731 2634 254 531 3723

# CLASSES m µ M σ 98 220 254 66 93 184 256 36 0 179 221 30 0 72 157 38 42 75 109 17 0 22 47 10 3 25 43 10 0 176 859 252 9 606 647 181 45 60 108 15 11 148 184 25 47 58 75 8 228 260 336 27 0 145 189 53 0 28 36 13 13 55 82 17 66 161 308 80 107 124 641 29 20 22 23 1 48 142 144 42 23 41 77 16

% CLASSES INV m µ M σ 0.48 0.54 0.63 0.05 0.49 0.66 0.69 0.04 0 0.07 0.08 0.01 0 0.12 0.29 0.04 0.5 0.81 0.87 0.11 0 0.18 0.38 0.1 0 0.52 0.57 0.09 0 0.63 0.8 0.13 0.24 0.34 0.58 0.06 0.01 0.04 0.06 0.01 0.04 0.47 0.52 0.06 0.35 0.38 0.42 0.02 0 0.02 0.04 0.01 0 0.42 0.44 0.08 0 0 0 0 0 0 0.03 0 0.11 0.17 0.26 0.05 0.12 0.47 0.62 0.04 0.04 0.09 0.09 0.01 0.08 0.1 0.11 0.01 0 0.06 0.1 0.03 m 524 1377 0 0 245 0 1 0 80 163 25 684 1075 0 0 274 304 815 154 798 335

# ROUTINES µ M 2395 2946 5214 7542 5300 8360 271 1304 1421 2900 346 630 118 221 2326 113813 7032 7894 214 309 886 1515 790 941 1263 1863 1448 2686 67 100 680 1199 746 3819 980 1761 171 173 952 959 612 1466 σ 1005 1326 2064 308 696 87 62 35183 2249 52 309 51 179 777 39 200 1130 176 6 66 355

Table 8. Project specification statistics for flattened classes.

LOC µ M 52958 65625 45200 61922 27498 40750 8457 53316 13069 21941 4131 10840 3591 6131 24146 163552 83444 88284 8358 20602 21700 29700 12141 14540 30343 40820 41616 55932 1741 2352 9550 13294 42055 73760 24993 35013 5751 5963 10688 10872 7549 15538

% ROUTINES SPEC m µ M σ 0.58 0.6 0.77 0.06 0.54 0.68 0.69 0.03 0 0.2 0.25 0.03 0 0.84 1 0.15 0.7 0.83 0.91 0.04 0 0.85 0.88 0.08 0 0.91 0.99 0.12 0 0.67 0.77 0.09 0.63 0.65 0.8 0.05 0 0.03 0.05 0.01 0.16 0.62 0.72 0.09 0.31 0.49 0.53 0.06 0 0.16 0.22 0.07 0 0.77 0.98 0.1 0 0.09 0.4 0.07 0.02 0.11 0.18 0.02 0.17 0.42 0.76 0.21 0.37 0.45 0.74 0.12 0.32 0.49 0.54 0.04 0.04 0.69 0.7 0.31 0 0.03 0.06 0.02

% ROUTINES PRE m µ M σ 0.37 0.39 0.48 0.04 0.38 0.48 0.49 0.02 0 0.11 0.18 0.03 0 0.49 0.6 0.16 0.51 0.57 0.74 0.05 0 0.46 0.53 0.05 0 0.5 0.83 0.09 0 0.49 0.67 0.08 0.5 0.5 0.74 0.06 0 0.03 0.04 0.01 0.16 0.46 0.53 0.06 0.19 0.37 0.43 0.06 0 0.15 0.21 0.07 0 0.69 0.95 0.11 0 0.08 0.4 0.07 0.02 0.11 0.18 0.02 0.13 0.25 0.68 0.2 0.3 0.38 0.67 0.11 0.14 0.33 0.35 0.04 0.02 0.04 0.05 0.01 0 0.02 0.05 0.02

% ROUTINES POST m µ M σ 0.38 0.41 0.59 0.06 0.3 0.45 0.49 0.03 0 0.1 0.13 0.02 0 0.62 1 0.18 0.53 0.61 0.76 0.06 0 0.69 0.73 0.07 0 0.68 0.72 0.12 0 0.4 0.52 0.11 0.08 0.43 0.48 0.06 0 0 0.01 0 0 0.27 0.35 0.06 0.12 0.16 0.19 0.02 0 0.01 0.02 0.01 0 0.72 0.86 0.08 0 0.01 0.07 0.01 0.01 0.08 0.18 0.03 0.08 0.14 0.35 0.04 0.18 0.22 0.31 0.03 0.21 0.28 0.33 0.02 0.04 0.67 0.67 0.3 0 0.01 0.02 0.01

−A inv weak vs same

−A inv strong vs same

2.90E+01 1.44E-02 -2.00E+00 -4.19E-01 2.90E+01 1.44E-02 -2.00E+00 -4.19E-01

+A inv strong vs same

post weak vs strong

1.90E+01 3.78E-02 0.00E+00 -4.75E-01 1.90E+01 3.78E-02 0.00E+00 -4.75E-01

+A inv weak vs same

NE pre weak vs strong

3.15E+01 1.95E-02 -2.00E+00 -4.04E-01 3.15E+01 1.95E-02 -2.00E+00 -4.04E-01

NE inv weak vs strong

pre weak vs strong

2.90E+01 4.78E-03 -1.10E+01 -5.03E-01 2.90E+01 4.78E-03 -1.10E+01 -5.03E-01

inv weak vs strong

NE chang vs unch spec

0.00E+00 9.54E-07 -6.80E+01 -9.96E-01 0.00E+00 9.54E-07 -6.80E+01 -9.96E-01

NE post weak vs strong

chang vs unch spec V p ∆(µ) d V p ∆(µ) d

0.00E+00 6.15E+01 1.82E-01 0.00E+00 -1.00E+00 0.00E+00 -1.82E-01 0.00E+00 6.15E+01 1.82E-01 0.00E+00 -1.00E+00 0.00E+00 -1.82E-01

5.85E+01 6.40E-01 0.00E+00 -1.29E-01 5.85E+01 6.40E-01 0.00E+00 -1.29E-01

0.00E+00 1.41E-02 0.00E+00 -6.27E-01 0.00E+00 1.41E-02 0.00E+00 -6.27E-01

0.00E+00 1.40E-02 0.00E+00 -4.17E-01 0.00E+00 1.40E-02 0.00E+00 -4.17E-01

3.00E+00 1.41E-01 0.00E+00 -3.09E-01 3.00E+00 1.41E-01 0.00E+00 -3.09E-01

0.00E+00 2.23E-02 0.00E+00 -5.25E-01 0.00E+00 2.23E-02 0.00E+00 -5.25E-01

−A inv weak vs same

−A inv strong vs same

5.00E+01 4.15E-02 -4.00E+00 -3.20E-01 5.00E+01 4.15E-02 -4.00E+00 -3.20E-01

+A inv strong vs same

2.55E+01 9.60E-02 0.00E+00 -5.85E-01 2.55E+01 9.60E-02 0.00E+00 -5.85E-01

+A inv weak vs same

post weak vs strong

5.45E+01 1.07E-01 -4.00E+00 -2.83E-01 5.45E+01 1.07E-01 -4.00E+00 -2.83E-01

NE inv weak vs strong

NE pre weak vs strong

7.75E+01 1.92E-01 -8.00E+00 -2.61E-01 7.75E+01 1.92E-01 -8.00E+00 -2.61E-01

inv weak vs strong

pre weak vs strong

0.00E+00 9.54E-07 -2.28E+02 -8.68E-01 0.00E+00 9.54E-07 -2.28E+02 -8.68E-01

NE post weak vs strong

NE chang vs unch spec

V p ∆(µ) d V p ∆(µ) d

chang vs unch spec

Table 9. Change analysis with majority for non-flattened (top) and flattened (bottom) classes.

0.00E+00 4.35E+01 7.00E-02 0.00E+00 -1.00E+00 0.00E+00 -3.32E-01 0.00E+00 4.35E+01 7.00E-02 0.00E+00 -1.00E+00 0.00E+00 -3.32E-01

4.80E+01 5.12E-01 0.00E+00 -1.21E-01 4.80E+01 5.12E-01 0.00E+00 -1.21E-01

0.00E+00 1.43E-02 0.00E+00 -5.91E-01 0.00E+00 1.43E-02 0.00E+00 -5.91E-01

0.00E+00 1.41E-02 0.00E+00 -4.11E-01 0.00E+00 1.41E-02 0.00E+00 -4.11E-01

3.50E+00 9.04E-02 0.00E+00 -3.34E-01 3.50E+00 9.04E-02 0.00E+00 -3.34E-01

0.00E+00 2.23E-02 0.00E+00 -5.37E-01 0.00E+00 2.23E-02 0.00E+00 -5.37E-01

Table 10. Change analysis for non-flattened (top) and flattened (bottom) classes.

31

32

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Overall

τ 0.98 0.27 0.66 0.86 0.13 0.81 -0.10 0.77 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.64

0.00E+00 0.00E+00 0.00E+00 4.36E-09 6.15E-11 0.00E+00 0.00E+00

p 0.00E+00 0.00E+00 4.75E-12 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 3.65E-08 0.00E+00 0.00E+00 0.00E+00 0.00E+00 4.20E-11 0.00E+00 0.00E+00 0.00E+00 0.00E+00 9.32E-12 2.07E-08 0.00E+00 0.00E+00

τ 0.60 0.62 0.34 0.89 0.38 0.58 0.80 0.78 -0.14 0.97 0.45 0.97 1.00 0.38 1.00 0.96 0.98 0.89 1.00 0.79 1.00 0.82 0.00E+00 0.00E+00 3.22E-12 2.12E-09 0.00E+00 0.00E+00

p 0.00E+00 0.00E+00 1.24E-10 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 2.62E-14 0.33 0.66 1.00 0.98 1.00 0.75

τ 0.75 0.38 0.31 0.36 0.70 0.89 1.00 0.76 0.29 1.00 0.78 0.94 1.00 0.92 1.00

0.00E+00 1.80E-02 0.00E+00 1.21E-07 9.76E-09 0.00E+00 0.00E+00

p 1.55E-04 0.00E+00 0.00E+00 0.00E+00 0.00E+00 1.41E-06 0.00E+00 0.00E+00 0.00E+00 0.00E+00 4.52E-06 4.19E-02 0.00E+00 0.00E+00

1.00 -0.06 0.91 1.00 1.00 0.80 0.40

τ 0.16 0.46 0.95 0.97 0.45 0.36 0.90 0.79 0.71 1.00 -0.18 -0.22 1.00 1.00

CLASSES W / INV AVG PRE STRENGTH AVG POST STRENGTH AVG INV STRENGTH

p 0.00E+00 0.00E+00 0.00E+00 0.00E+00 1.44E-03 0.00E+00 4.19E-02 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00

%

Table 11. Correlation of measures in flat and non-flat classes.

τ 0.65 0.17 -0.01 0.60 0.01 -0.05 0.92 0.28 0.91 0.98 0.90 0.64 0.99 0.54 0.99 0.56 0.89 0.97 1.00 0.97 0.79 0.53

ROUTINES W / SPEC

p 0.00E+00 0.00E+00 8.98E-01 0.00E+00 8.88E-01 4.28E-01 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 9.79E-10 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 5.70E-12 3.43E-12 0.00E+00 0.00E+00

%

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Overall

% SPEC /# ROUTINES p τ 1.02E-94 -0.82 0.00E+00 0.66 1.17E-10 0.31 9.98E-04 -0.09 3.04E-05 0.17 2.16E-04 -0.24 1.94E-32 -0.56 5.77E-74 -0.40 0.00E+00 -0.95 0.00E+00 0.67 4.77E-100 -0.81 4.64E-01 -0.08 1.89E-06 0.17 4.76E-41 -0.76 1.44E-03 0.32 7.14E-62 -0.59 0.00E+00 0.54 0.00E+00 0.47 5.01E-01 0.10 7.35E-06 0.63 1.04E-02 0.15 1.15E-05 0.03

# SPEC /# ROUTINES p τ 0.00E+00 0.93 0.00E+00 0.98 0.00E+00 0.98 0.00E+00 0.97 0.00E+00 0.53 0.00E+00 0.91 0.00E+00 0.99 0.00E+00 0.97 0.00E+00 0.56 0.00E+00 0.76 0.00E+00 0.97 6.07E-06 0.48 0.00E+00 0.48 0.00E+00 0.97 4.44E-16 0.84 0.00E+00 0.82 0.00E+00 0.93 0.00E+00 0.81 3.87E-02 0.32 4.36E-09 0.84 0.00E+00 0.64 0.00E+00 0.76

% INV /# CLASSES p τ 1.21E-117 -0.93 1.13E-08 0.11 8.25E-04 -0.17 0.00E+00 0.35 5.69E-07 -0.21 6.16E-13 0.51 0.00E+00 0.60 0.00E+00 0.57 8.94E-209 -0.77 9.32E-01 0.01 1.94E-88 -0.77 5.23E-08 0.58 0.00E+00 0.63 4.32E-03 0.17

# INV /# CLASSES p τ 0.00E+00 0.76 0.00E+00 0.95 0.00E+00 0.85 0.00E+00 0.92 0.00E+00 0.70 0.00E+00 0.91 0.00E+00 0.95 0.00E+00 0.94 0.00E+00 0.28 2.90E-12 0.58 0.00E+00 0.65 8.88E-16 0.88 0.00E+00 0.83 0.00E+00 0.96

2.32E-01 3.55E-279 2.10E-95 6.06E-02 4.43E-01 9.82E-11 0.00E+00

2.32E-01 0.00E+00 1.66E-07 5.77E-03 6.61E-05 0.00E+00 0.00E+00

-0.05 -0.83 -0.76 -0.33 -0.12 0.38 0.20

-0.05 0.88 0.20 0.51 0.64 0.85 0.61

Table 12. Correlation of measures of specification in non-flat classes.

33

AVG /% PRE

p 0.00E+00 0.00E+00 1.69E-01 2.71E-05 4.23E-04 1.55E-05 0.00E+00 3.65E-03 6.75E-32 2.88E-10 0.00E+00 7.32E-14 6.66E-02 1.67E-11 0.00E+00 5.85E-01 2.76E-212 3.93E-63 5.90E-03 2.76E-10 0.00E+00 0.00E+00

τ 0.65 0.44 0.07 -0.11 0.15 0.30 0.54 0.07 -0.30 0.48 0.32 -0.81 0.07 -0.38 0.87 -0.02 -0.72 -0.59 -0.40 -0.88 0.75 0.32

AVG /% POST

p 0.00E+00 0.00E+00 4.94E-01 1.42E-01 0.00E+00 7.87E-10 0.00E+00 0.00E+00 1.85E-05 0.00E+00 0.00E+00 1.46E-03 0.00E+00 1.86E-33 4.23E-11

τ 0.43 0.26 0.03 -0.04 0.52 0.41 0.58 0.51 0.12 0.91 0.53 0.34 0.79 -0.70 0.77

4.73E-07 3.50E-21 9.38E-01 3.59E-04 0.00E+00 1.23E-12

0.12 -0.34 0.01 -0.54 0.71 0.05

AVG /% INV

p 0.00E+00 0.00E+00 5.23E-01 6.90E-11 6.58E-01 6.79E-02 2.83E-13 0.00E+00 0.00E+00 4.31E-04 6.67E-13 1.15E-08 0.00E+00 5.77E-01

τ 0.78 0.19 -0.04 0.18 0.02 0.13 0.36 0.57 0.57 -0.29 0.29 0.62 0.54 0.03

0.00E+00 0.00E+00 0.00E+00 5.09E-01 1.68E-04 0.00E+00 0.00E+00

1.00 0.47 0.63 -0.12 -0.61 0.57 0.47

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Overall

% SPEC /# ROUTINES p τ 1.28E-70 -0.70 8.85E-06 -0.08 1.51E-14 0.37 0.00E+00 0.29 6.58E-01 0.02 2.41E-01 0.08 2.47E-34 -0.58 0.00E+00 0.20 8.05E-274 -0.87 0.00E+00 0.64 9.03E-93 -0.77 8.77E-06 -0.46 1.92E-06 0.17 3.92E-09 -0.33 2.08E-03 0.31 9.07E-36 -0.44 0.00E+00 0.64 0.00E+00 0.38 5.01E-01 0.10 9.35E-06 0.62 6.22E-10 0.36 9.70E-03 0.02

# SPEC /# ROUTINES p τ 0.00E+00 0.92 0.00E+00 0.92 0.00E+00 0.98 0.00E+00 0.99 0.00E+00 0.98 0.00E+00 0.98 0.00E+00 0.99 0.00E+00 0.98 0.00E+00 0.89 0.00E+00 0.75 0.00E+00 0.97 9.08E-03 0.28 0.00E+00 0.48 0.00E+00 0.97 4.44E-16 0.84 0.00E+00 0.78 0.00E+00 0.93 0.00E+00 0.80 3.87E-02 0.32 2.99E-09 0.85 0.00E+00 0.84 0.00E+00 0.80

% INV /# CLASSES p τ 3.09E-117 -0.93 9.35E-35 -0.23 8.13E-19 -0.44 0.00E+00 0.47 3.63E-13 0.30 2.22E-16 0.58 3.70E-04 -0.17 0.00E+00 0.69 8.94E-209 -0.77 9.32E-01 0.01 1.94E-88 -0.77 5.23E-08 0.58 0.00E+00 0.63 4.32E-03 0.17

# INV /# CLASSES p τ 0.00E+00 0.88 0.00E+00 0.94 0.00E+00 0.89 0.00E+00 0.92 0.00E+00 0.72 0.00E+00 0.88 0.00E+00 0.97 0.00E+00 0.95 0.00E+00 0.28 2.90E-12 0.58 0.00E+00 0.65 8.88E-16 0.88 0.00E+00 0.83 0.00E+00 0.96

2.32E-01 3.55E-279 2.10E-95 6.06E-02 4.43E-01 9.82E-11 0.00E+00

2.32E-01 0.00E+00 1.66E-07 5.77E-03 6.61E-05 0.00E+00 0.00E+00

-0.05 -0.83 -0.76 -0.33 -0.12 0.38 0.17

-0.05 0.88 0.20 0.51 0.64 0.85 0.59

Table 13. Correlation of measures of specification in flat classes.

34

AVG /% PRE

p 0.00E+00 0.00E+00 0.00E+00 7.60E-05 0.00E+00 4.86E-05 8.88E-16 4.88E-15 0.00E+00 3.48E-11 0.00E+00 1.48E-16 7.10E-02 3.79E-09 0.00E+00 2.22E-14 4.87E-196 7.29E-65 5.90E-03 1.14E-06 0.00E+00 0.00E+00

τ 0.51 0.33 0.50 -0.10 0.53 0.27 0.39 0.17 0.51 0.50 0.60 -0.89 0.06 -0.34 0.88 -0.31 -0.69 -0.59 -0.40 -0.68 0.75 0.20

AVG /% POST

p 0.00E+00 7.64E-24 4.93E-07 3.25E-64 8.23E-43 1.87E-02 2.66E-05 2.95E-10 4.31E-01 0.00E+00 0.00E+00 2.58E-03 0.00E+00 9.97E-11 4.15E-11

τ 0.38 -0.19 0.25 -0.45 -0.56 0.16 0.21 0.14 0.02 0.92 0.51 0.32 0.80 -0.37 0.78

0.00E+00 1.50E-08 9.38E-01 2.12E-03 0.00E+00 0.00E+00

0.22 -0.20 0.01 -0.47 0.72 0.25

AVG /% INV

p 1.42E-01 0.00E+00 6.44E-13 4.99E-11 5.37E-14 0.00E+00 1.36E-03 0.00E+00 0.00E+00 4.31E-04 4.52E-80 1.00E-06 0.00E+00 5.77E-01

τ -0.06 0.16 -0.39 0.18 0.31 0.80 0.16 0.69 0.42 -0.29 -0.73 -0.53 0.54 0.03

0.00E+00 3.26E-114 0.00E+00 5.09E-01 1.68E-04 3.13E-09 0.00E+00

1.00 -0.54 0.64 -0.12 -0.61 0.36 0.57

% Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Overall

PRE PUBLIC / PRIVATE

p 0.00E+00 0.00E+00 2.16E-09 1.17E-05 9.01E-27 1.31E-03 1.59E-02 1.57E-07 0.00E+00 0.00E+00 0.00E+00 1.66E-09 1.98E-07 7.66E-05 6.28E-09 1.97E-02 0.00E+00 0.00E+00 2.01E-08 5.83E-02 0.00E+00 0.00E+00

τ 0.76 0.73 0.29 -0.12 -0.44 -0.22 0.12 -0.12 0.94 0.84 0.36 -0.63 0.19 0.23 0.59 -0.08 0.40 0.33 0.84 0.26 0.59 0.38

%

POST PUBLIC / PRIVATE

p 0.00E+00 0.00E+00 0.00E+00 8.31E-01 1.05E-36 2.35E-07 0.00E+00 2.08E-02 0.00E+00 0.00E+00 0.00E+00 4.42E-03 0.00E+00 5.46E-06 4.78E-05

τ 0.51 0.57 0.49 -0.01 -0.52 0.34 0.65 0.05 0.73 0.80 0.86 0.30 0.80 0.26 0.42

7.38E-12 1.56E-11 1.45E-01 2.22E-06 0.00E+00 0.00E+00

0.16 0.24 0.21 0.67 0.78 0.38

%

SPEC PUBLIC / PRIVATE

p 0.00E+00 0.00E+00 4.18E-01 1.48E-39 3.27E-30 7.48E-05 3.59E-02 2.66E-26 0.00E+00 0.00E+00 0.00E+00 2.20E-06 1.45E-07 2.95E-03 3.11E-04 1.86E-02 0.00E+00 0.00E+00 1.55E-08 7.73E-07 0.00E+00 0.00E+00

Table 14. Correlation of measures between public and private routines in non-flat classes.

35

τ 0.75 0.64 0.04 -0.36 -0.47 -0.26 0.10 -0.24 0.94 0.85 0.66 -0.50 0.19 0.17 0.37 -0.08 0.46 0.51 0.82 0.68 0.81 0.39

% Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra Overall

PRE PUBLIC / PRIVATE

p 0.00E+00 0.00E+00 1.00E-06 7.45E-02 3.17E-01 1.85E-03 1.39E-01 0.00E+00 0.00E+00 0.00E+00 0.00E+00 2.89E-11 4.07E-08 1.01E-04 9.05E-09 1.10E-44 0.00E+00 0.00E+00 2.01E-08 1.12E-01 0.00E+00 0.00E+00

τ 0.66 0.41 0.24 0.05 -0.04 -0.21 0.07 0.37 0.68 0.83 0.40 -0.69 0.20 0.22 0.59 -0.50 0.50 0.60 0.84 0.22 0.87 0.35

%

POST PUBLIC / PRIVATE

p 0.00E+00 5.13E-18 0.00E+00 1.07E-08 0.00E+00 3.49E-01 2.84E-04 2.68E-01 0.00E+00 0.00E+00 0.00E+00 2.37E-02 0.00E+00 0.00E+00 4.59E-05

τ 0.61 -0.16 0.60 -0.15 0.54 -0.06 0.18 0.02 0.27 0.80 0.80 0.23 0.81 0.47 0.43

5.61E-06 5.17E-04 1.45E-01 6.58E-07 0.00E+00 0.00E+00

0.11 0.12 0.21 0.70 0.77 0.50

%

SPEC PUBLIC / PRIVATE

p 0.00E+00 8.27E-01 2.25E-10 3.82E-07 8.73E-06 2.13E-06 1.19E-02 0.00E+00 0.00E+00 0.00E+00 0.00E+00 3.94E-12 4.19E-09 4.77E-04 7.61E-04 4.53E-43 0.00E+00 0.00E+00 3.78E-08 3.65E-07 0.00E+00 0.00E+00

Table 15. Correlation of measures between public and private routines in flat classes.

36

τ 0.76 0.00 0.31 -0.14 -0.18 0.32 0.12 0.19 0.84 0.83 0.66 -0.72 0.21 0.20 0.34 -0.49 0.55 0.59 0.80 0.70 0.90 0.42

Project AutoTest EiffelBase EiffelProgramAnalysis GoboKernel GoboStructure GoboTime GoboUtility GoboXML Boogie CCI Dafny Labs Quickgraph Rxx shweet DirectVCGen ESCJava JavaFE Logging RCC Umbra

% PRE VOID % PRE ∀, ∃ m µ M σ m µ M σ 0.67 0.69 0.76 0.02 0 0 0 0 0.23 0.36 0.46 0.05 0 0 0 0 0 0.62 0.67 0.08 0 0 0 0 0 0.56 1 0.19 0 0 0.01 0 0.35 0.43 0.75 0.12 0 0 0 0 0 0.65 0.8 0.08 0 0 0 0 0 0.93 1 0.12 0 0 0 0 0 0.64 0.75 0.05 0 0 0 0 0.94 0.96 0.99 0.01 0 0.01 0.01 0 0 0.8 0.92 0.32 0 0 0 0 0.8 0.96 0.98 0.02 0 0 0 0 0.93 0.96 0.97 0.01 0 0 0 0 0 0.94 0.99 0.34 0 0.01 0.07 0.02 0 0.95 1 0.08 0 0 0 0 0 0.83 1 0.4 0 0 0 0 1 1 1 0 0 0 0 0 0.85 0.93 0.99 0.05 0 0 0 0 0.92 0.94 0.98 0.02 0 0 0.01 0 0.88 0.95 0.95 0.02 0 0 0 0 0.71 0.88 0.9 0.08 0 0 0 0 0 0.25 0.33 0.16 0 0 0 0

% POST VOID % POST ∀, ∃ m µ M σ m µ M σ 0.35 0.38 0.64 0.12 0 0.01 0.02 0.01 0.06 0.14 0.25 0.06 0 0 0 0 0 0.19 0.22 0.02 0 0 0 0 0 0.47 1 0.16 0 0 0 0 0.03 0.07 0.1 0.02 0 0 0 0 0 0.29 0.53 0.11 0 0 0 0 0 0.62 0.89 0.15 0 0 0 0 0 0.37 0.61 0.06 0 0 0 0 0.72 0.9 1 0.04 0 0 0.01 0 0 0 1 0.49 0 0 0 0 0 0.88 0.94 0.12 0 0 0 0 0.65 0.71 0.8 0.05 0 0 0 0 0 0.34 0.67 0.21 0 0 0 0 0 0.92 1 0.09 0 0 0 0 0 1 1 0.49 0 0 0 0 1 1 1 0 0 0 0 0 0.45 0.69 0.88 0.11 0 0.01 0.03 0.01 0.65 0.69 0.84 0.05 0 0 0 0 0 0.28 0.43 0.1 0 0 0 0 0.16 0.89 0.94 0.36 0 0 0 0 0 0 0 0 0 0 0 0

% POST OLD m µ M σ 0 0.02 0.02 0.01 0.01 0.13 0.2 0.02 0 0.02 0.02 0.01 0 0.05 0.2 0.04 0.07 0.39 0.52 0.13 0 0.29 0.32 0.06 0 0 0.02 0.01 0 0.01 0.03 0.01 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0.03 0 0 1 0.31 0 0 0.01 0.01 0 0 0 0 0 0 0 0 0 0 0.02 0 0.02 0.03 0.04 0.01 0.02 0.02 0.03 0 0 0 0 0 0 0.67 0.67 0.32

% INV VOID % INV ∀, ∃ m µ M σ m µ M σ 0.85 0.86 0.98 0.04 0.02 0.02 0.02 0 0.08 0.24 0.29 0.06 0 0 0 0 0 0.14 0.22 0.08 0 0 0 0 0 0.73 1 0.21 0 0 0 0 0.33 0.68 0.78 0.13 0 0 0 0 0 0.5 0.6 0.17 0 0 0 0 0 0.82 1 0.14 0 0 0 0 0 0.85 0.94 0.04 0 0 0 0 1 1 1 0 0 0.06 0.07 0.02 1 1 1 0 0 0 0 0 0.98 1 1 0 0 0.01 0.02 0 0.92 0.97 1 0.03 0 0 0 0 0 0.2 0.36 0.11 0 0.1 0.17 0.06 0 1 1 0.12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.37 0.54 0.63 0.09 0 0.04 0.09 0.04 0.6 0.77 0.83 0.04 0.08 0.09 0.19 0.02 0 0 0 0 0 0 0 0 0.5 0.5 1 0.22 0 0 0 0 0 0 0 0 0 0.25 0.25 0.12

Table 16. Syntactic elements found in contracts (non-flattened classes).

37

11

Non-flat classes

The following pages show the plots for the 21 projects of our study, considering nonflat classes. The dotted lines, when present, mark mean values of the various quantities. The thin continuous lines that do not appear in the legend track the total number of routines, classes, or both, whose absolute values are scaled to the range determined by the main data represented in the graph. When two of such thin lines are present, the red one tracks the number of classes and the aquamarine one tracks the number of routines. When only one (red) thin line is present, it tracks the number of routines or classes according to the main content of the graph.

38

# routines

# routines

12

10

8

6

4

2

spec

body+spec 0 (nonempty 50 spec)100 body

revision

150

200

250

300

AutoTest: # changed routines (for nonempty spec)

150

revision

200

250

80

800

revision

600

1000

1200

spec

body+spec 0 (nonempty 20 spec) 40 body revision

60

80

100

120

GoboTime: # changed routines (for nonempty spec)

spec

body+spec 200 spec) 400 body0 (nonempty

EiffelBase: # changed routines (for nonempty spec)

revision

100

150

200

body+spec 0 (nonempty 50 body spec) spec

100

150

200

GoboUtility: # changed routines (for nonempty spec)

body+spec 0 (nonempty spec) 50 body spec

EiffelProgramAnalysis: # changed routines (for nonempty spec)

Fig. 17. Non flat: Changed routines (with nonempty unchanged spec) (Eiffel projects)

spec

body+spec 0 (nonempty 50 spec) 100 body

GoboStructure: # changed routines (for nonempty spec)

0

15

10

5

0

# routines # routines

60 40 20 0 8 6 4 2 0

6 5 4 3 2 1 0 10 8 6

revision

2

# routines # routines

8

0

4

# routines # routines

6 4 2 0 25 20 15 10 5 0

39

400 revision

300

500

600

body+spec body0 (nonempty200 spec) spec

revision

400

600

800

GoboXML: # changed routines (for nonempty spec)

spec

body+spec 0 (nonempty 100 spec) 200 body

GoboKernel: # changed routines (for nonempty spec)

# routines

150

100

50

0

120

100

80

400

revision

600

800

body+spec 0 (nonempty spec) 100 body spec

200

300

Quickgraph: # changed routines (for nonempty spec)

spec

body+spec 200 body (nonempty spec)

Boogie: # changed routines (for nonempty spec)

40 revision

60

80

body+spec 0 (nonempty spec) 50 body spec revision

100

150

100

Rxx: # changed routines (for nonempty spec)

body+spec 0 (nonempty 20spec) body spec

CCI: # changed routines (for nonempty spec)

Fig. 18. Non flat: Changed routines (with nonempty unchanged spec) (C# projects)

revision

40

20

# routines

2.0 1.5

14 12 10 8 6 4

1.0 0.5 0.0 150 100

0

60

# routines # routines 50 0

# routines # routines

250

revision

200

300

350

400

spec

body+spec 0 (nonempty 10 spec) 20 body

revision

30

40

50

60

shweet: # changed routines (for nonempty spec)

spec

body+spec 100 150 body (nonempty spec)

Dafny: # changed routines (for nonempty spec) 35 30 25 20 15 10 5 0

2 0 2.0 1.5 1.0 0.5 0.0

40

# routines

spec

body+spec 0 (nonempty 10 spec) body

20 revision

30

40

50

Labs: # changed routines (for nonempty spec)

# routines

# routines

revision

200

300

400

15

revision

20

25

30

body+spec 0 (nonempty 5 body spec)10 spec

revision

400

600

800

revision

15

20

25

30

RCC: # changed routines (for nonempty spec)

body+spec 0 (nonempty200 body spec) spec

ESCJava: # changed routines (for nonempty spec)

Fig. 19. Non flat: Changed routines (with nonempty unchanged spec) (Java projects)

spec

body+spec 0 (nonempty 5 spec)10 body

Logging: # changed routines (for nonempty spec)

body+spec 0 (nonempty 100 body spec) spec

DirectVCGen: # changed routines (for nonempty spec) 300 250 200

300 250 200 150

25

20

15

10

5

0

30

25

20

15

10

5

0

# routines # routines

100 50 0 200 150 100 50 0

# routines # routines

150 100 50 0 1.0 0.8 0.6 0.4 0.2 0.0

41

200 revision

300

400

body+spec 0 (nonempty spec)50 body spec

revision

100

150

Umbra: # changed routines (for nonempty spec)

body+spec 0 (nonempty spec) 100 body spec

JavaFE: # changed routines (for nonempty spec)

# routines

6

5

4

3

2

1

100

150

revision

200

250

300

0 body+spec spec

50

100

revision

150

200

250

GoboStructure: # changed routines (w/ or w/o body change)

0 50 body+spec spec

AutoTest: # changed routines (w/ or w/o body change) 80

400

800

revision

600

1000

1200

0 20 body+spec spec

40 revision

60

80

100

120

GoboTime: # changed routines (w/ or w/o body change)

0 200 body+spec spec

EiffelBase: # changed routines (w/ or w/o body change)

50

100 revision

150

200

0 body+spec spec

50

100

150

200

GoboUtility: # changed routines (w/ or w/o body change)

0 body+spec spec

revision

200

400 revision

300

500

600

0 body+spec spec

200

revision

400

600

800

GoboXML: # changed routines (w/ or w/o body change)

0 100 body+spec spec

EiffelProgramAnalysis: # changed routines (w/ or w/o body change) GoboKernel: # changed routines (w/ or w/o body change)

Fig. 20. Non-flat: Routines with changed specification (with or without body changing) (Eiffel projects)

# routines

0

15

10

5

0

# routines # routines

60 40 20 0 4 3 2 1 0

6 5 4 3 2 1 0 10 8 6 2

# routines # routines

8

0

4

# routines # routines

6 4 2 0 25 20 15 10 5 0

42

# routines

150

100

50

200 body+spec spec

400

revision

600

800

Boogie: # changed routines (w/ or w/o body change)

0 body+spec spec

100

200

300

Quickgraph: # changed routines (w/ or w/o body change)

0

120

100

80

60

20

40 revision

60

80

100

0 body+spec spec

50 revision

100

150

Rxx: # changed routines (w/ or w/o body change)

0 body+spec spec

CCI: # changed routines (w/ or w/o body change)

150

250

revision

200

300

350

400

0 10 body+spec spec

20

revision

30

40

50

60

shweet: # changed routines (w/ or w/o body change)

100 body+spec spec

Dafny: # changed routines (w/ or w/o body change)

Fig. 21. Non-flat: Routines with changed specification (with or without body changing) (C# projects)

revision

20

# routines

2.0 1.5

0

40

# routines # routines

14 12 10 8 6 4

1.0 0.5 0.0 14 12 10 8 6 4 2 0

# routines # routines

35 30 25 20 15 10 5 0

2 0 2.0 1.5 1.0 0.5 0.0

43

# routines

0 body+spec spec

10

20 revision

30

40

50

Labs: # changed routines (w/ or w/o body change)

# routines

100

200

revision

300

400

0 5 body+spec spec

10

revision

15

20

25

30

Logging: # changed routines (w/ or w/o body change)

0 body+spec spec

DirectVCGen: # changed routines (w/ or w/o body change)

8

6

4

2

0

300 250 200

200 revision

400

600

800

0 5 body+spec spec

10 revision

15

20

25

30

RCC: # changed routines (w/ or w/o body change)

0 body+spec spec

ESCJava: # changed routines (w/ or w/o body change) 300 250 200

100

200 revision

300

400

0 body+spec spec

50 revision

100

150

Umbra: # changed routines (w/ or w/o body change)

0 body+spec spec

JavaFE: # changed routines (w/ or w/o body change)

Fig. 22. Non-flat: Routines with changed specification (with or without body changing) (Java projects)

# routines

30

25

20

15

10

5

0

# routines # routines

150 100 50 0 200 150 100 50 0

# routines # routines

150 100 50 0 1.0 0.8 0.6 0.4 0.2 0.0

44

# routines

# routines

revision

150

200

250

300

spec

revision

150

200

250

800

revision

600

1000

1200

spec

body+spec body 0 (nonempty 20 spec) 40 body revision

60

80

100

120

GoboTime: # changed routines (types of change)

spec

body+spec body 200 spec) 400 body0 (nonempty

EiffelBase: # changed routines (types of change)

Fig. 23. Non-flat: changed routines (types of change) (Eiffel projects)

body+spec body 0 (nonempty 50 spec) 100 body

GoboStructure: # changed routines (types of change)

spec

body+spec body 0 (nonempty 50 spec)100 body

AutoTest: # changed routines (types of change) 250 200

35

30

25

20

15

10

5

0

50

40

30

20

10

0

# routines # routines

150 100 50 0 15 10 5 0

# routines # routines

revision

100

150

200

body+spec body 0 (nonempty 50 body spec) spec

revision

100

150

200

GoboUtility: # changed routines (types of change)

body+spec body 0 (nonempty spec) 50 body spec

EiffelProgramAnalysis: # changed routines (types of change) 15 10 5 0 10 8 6 4 2 0

# routines # routines

40 30 20 10 0 120 100 80 60 40 20 0

45

400 revision

300

500

600

body+spec body body0 (nonempty200 spec) spec

revision

400

600

800

GoboXML: # changed routines (types of change)

spec

body+spec body 0 (nonempty 100 spec) 200 body

GoboKernel: # changed routines (types of change)

150

100

50

0

120

100

80

# routines

# routines

400

revision

600

800

revision

200

300

40 revision

60

80

body+spec body 0 (nonempty spec) 50 body spec revision

100

Rxx: # changed routines (types of change)

body+spec body 0 (nonempty 20spec) body spec

CCI: # changed routines (types of change)

Fig. 24. Non-flat: changed routines (types of change) (C# projects)

body+spec body 0 (nonempty spec) 100 body spec

Quickgraph: # changed routines (types of change)

spec

body+spec body 200 body (nonempty spec)

Boogie: # changed routines (types of change) 10 8 6 4 2 0 150 100

150

100

50

# routines # routines

20 15 10 spec

250

revision

200

300

350

revision

30

40

50

shweet: # changed routines (types of change)

body+spec body 0 (nonempty 10 spec) 20 body

spec

body+spec body 100 150 body (nonempty spec)

Dafny: # changed routines (types of change)

60

400

50 40 30 20 10 0

5 0 15 10

0

60

40

20

0

# routines # routines 5 0

46

# routines

spec

body+spec body 0 (nonempty 10 spec) body

20 revision

30

40

Labs: # changed routines (types of change)

50

# routines

# routines

revision

200

300

spec

revision

15

20

25

30

400

body+spec body 0 (nonempty 5 body spec)10 spec

revision

400

600

800

revision

15

20

25

RCC: # changed routines (types of change)

body+spec body 0 (nonempty200 body spec) spec

30

ESCJava: # changed routines (types of change)

Fig. 25. Non-flat: changed routines (types of change) (Java projects)

body+spec body 0 (nonempty 5 spec)10 body

Logging: # changed routines (types of change)

body+spec body 0 (nonempty 100 body spec) spec

DirectVCGen: # changed routines (types of change) 300 250 200

300 250 200

80

60

40

20

0

30

25

20

15

10

5

0

# routines # routines

150 100 50 0 200 150 100 50 0

# routines # routines

150 100 50 0 25 20 15 10 5 0

47

200 revision

300

body+spec body 0 (nonempty spec)50 body spec

revision

100

400

150

Umbra: # changed routines (types of change)

body+spec body 0 (nonempty spec) 100 body spec

JavaFE: # changed routines (types of change)

3.5

3.0

2.5

2.0

1.5

1.0

0.5

0.0

3.5

3.0

2.5

2.0

1.5

100

revision

150

200

250

% inv mean inv # inv0 weak 50 # inv strong

100

150

200

250

GoboStructure: Average and % inv

% inv mean inv # inv0 weak 50 # inv strong

AutoTest: Average and % inv

300

400

800

revision

600

1000

40 revision

60

80

1200

100

GoboTime: Average and % inv

% inv mean inv 0 weak 20 # inv # inv strong

% inv mean inv # inv0 weak 200 # inv strong

EiffelBase: Average and % inv

120

% inv mean inv 0 weak # inv # inv strong

revision

100

150

50

100

150

GoboUtility: Average and % inv

50

200

200

EiffelProgramAnalysis: Average and % inv

% inv mean inv 0 weak # inv # inv strong

Fig. 26. Non-flat: Average invariant strength and % of classes with invariant (Eiffel projects)

revision

0.5

0.0

1.0

12 10 8 6 4 2 0 3.0 2.5 2.0 1.5 1.0 0.5 0.0

1.2 1.0 0.8 0.6 0.4 0.2 0.0 6 5 4 3

revision

1 0

2

2.0 1.5 1.0 0.5 0.0 12 10 8 6 4 2 0

48

% inv mean inv # inv0 weak # inv strong

% inv mean inv # inv0 weak 100 # inv strong

400 revision

300

500

200

revision

400

600

GoboXML: Average and % inv

200

600

800

GoboKernel: Average and % inv

35

30

25

20

15

10

5

0

5

4

3

2

1

% inv mean inv # inv0 weak # inv strong

# inv strong

% inv mean inv 200 # inv weak

revision

600

100

revision

200

300

Quickgraph: Average and % inv

400

Boogie: Average and % inv

800

% inv mean inv 0 weak # inv # inv strong

% inv mean inv 0 weak # inv # inv strong 20 revision

60

50 revision

100

Rxx: Average and % inv

40

CCI: Average and % inv

80

150

100

% inv mean inv 0 weak 10 # inv # inv strong

# inv strong

% inv mean inv 100 # inv weak

Fig. 27. Non-flat: Average invariant strength and % of classes with invariant (C# projects)

0

2.0 1.5 1.0 0.5 0.0 8 6 4 2 0

8 6

250

revision

200

300

20

revision

30

40

shweet: Average and % inv

150

Dafny: Average and % inv

50

350

60

400

8 6 4 2 0

4 2 0 1.0 0.5 0.0 −0.5 −1.0

49

% inv mean inv 0 weak # inv # inv strong

10

20 revision

30

Labs: Average and % inv

40

50

2.0

1.5

1.0

0.5

0.0

2.0

1.5

1.0

0.5

% inv mean inv # 0inv weak 5 # inv strong

% inv mean inv #0 inv weak # inv strong

revision

200

300

10

revision

15

20

Logging: Average and % inv

100

25

DirectVCGen: Average and % inv

30

400

% inv mean inv #0 inv weak 5 # inv strong

% inv mean inv 0 weak # inv # inv strong revision

400

600

10 revision

15

20

RCC: Average and % inv

200

ESCJava: Average and % inv

25

800

30

% inv mean inv 0 weak # inv # inv strong

% inv mean inv # inv0 weak # inv strong

Fig. 28. Non-flat: Average invariant strength and % of classes with invariant (Java projects)

0.0

3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 6 5 4 3 2 1 0

5 4 3 2 1 0 2.0 1.5 1.0 0.5 0.0

50

200 revision

300

50 revision

100

Umbra: Average and % inv

100

JavaFE: Average and % inv

150

400

100

revision

150

200

250

50

100

revision

150

200

250

GoboStructure: # classes and invariants

50

300

250 200 # inv

classes 0 w/ inv

# inv

classes 0 w/ inv 400

800

revision

600

1000

1200

20

40 revision

60

80

100

GoboTime: # classes and invariants

200

EiffelBase: # classes and invariants

120

Fig. 29. Non-flat: # of classes, classes with invariants, invariant clauses (Eiffel projects)

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv

AutoTest: # classes and invariants 200 150

300

250

200

150

100

50

# classes or invariants

# classes or invariants

100

80

60

40

20

# classes or invariants # classes or invariants

150 100 50 40 30 20 10 0

# classes or invariants # classes or invariants

100 50 0 40 30 20 10 0 # inv

classes 0 w/ inv

# inv

classes 0 w/ inv revision

100

150

50

revision

100

150

GoboUtility: # classes and invariants

50

200

200

EiffelProgramAnalysis: # classes and invariants

# classes or invariants # classes or invariants

150 100 50 0 800 600 400 200 0

51

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv

200

400 revision

300

500

600

200

revision

400

600

800

GoboXML: # classes and invariants

100

GoboKernel: # classes and invariants

revision

600

800

100

200

revision

300 # inv

classes 0 w/ inv

# inv

classes 0 w/ inv 40 revision

60

80

50 revision

100

Rxx: # classes and invariants

20

CCI: # classes and invariants

150

100

Fig. 30. Non-flat: # of classes, classes with invariants, invariant clauses (C# projects)

# inv

classes 0 w/ inv

400

Quickgraph: # classes and invariants

classes 200 w/ inv # inv

Boogie: # classes and invariants 100 80 60

600

500

400

300

200

100

0

# classes or invariants

# classes or invariants

300

250

200

150

100

50

0

# classes or invariants # classes or invariants

40 20 0 250 200 150 100 50 0

# classes or invariants # classes or invariants

classes w/ 0 inv # inv

150

250

revision

200

300

20

revision

30

40

350

50

shweet: # classes and invariants

10

classes w/ inv 100 # inv

Dafny: # classes and invariants

60

400

70 60 50 40 30 20

200 150 100 50 0 35 30 25 20 15 10 5 0

52

# classes or invariants

classes w/ 0inv # inv

10

20 revision

30

40

Labs: # classes and invariants

50

# classes or invariants

# classes or invariants

revision

200

300

5

10

revision

15

20

25

Logging: # classes and invariants

100

30

400

classes w/0inv # inv

# inv

classes 0 w/ inv

5

revision

400

600

10 revision

15

20

25

RCC: # classes and invariants

200

ESCJava: # classes and invariants

800

30

Fig. 31. Non-flat: # of classes, classes with invariants, invariant clauses (Java projects)

classes w/0inv # inv

classes w/0inv # inv

DirectVCGen: # classes and invariants 300 250 200 150

500 400 300

80

60

40

20

0

20

15

10

5

# classes or invariants # classes or invariants

100 50 80 60 40 20

# classes or invariants # classes or invariants

200 100 80 60 40 20 0

53 # inv

classes 0 w/ inv

# inv

classes 0 w/ inv 200 revision

300

50 revision

100

Umbra: # classes and invariants

100

JavaFE: # classes and invariants

150

400

# routines

# routines

500

50

100

revision

150

200

250

300

pre or post

pre post0

50

150

revision

200

250 pre or post

pre post0

200

400

800

revision

600

1000

1200

20

40 revision

60

80

100

120

GoboTime: Routines with some specification

pre or post

pre post0

EiffelBase: Routines with some specification

Fig. 32. Non-flat: # routines with specification (Eiffel projects)

100

GoboStructure: Routines with some specification

pre or post

pre post0

AutoTest: Routines with some specification

pre or post

pre post0

50 revision

100

150

200

50

revision

100

150

200

GoboUtility: Routines with some specification

pre or post

pre post0

EiffelProgramAnalysis: Routines with some specification

300 250 200

1200 1000 800 600

400

300

200

150

100

50

# routines # routines

400 200 200 150 100 50 0

# routines # routines

150 100 50 0 150 100 50 0

# routines # routines

300 200 100 0 1400 1000 800 600 400 200 0

54

pre or post

pre post0

100

200

400 revision

300

500

600

200

revision

400

600

800

GoboXML: Routines with some specification

pre or post

pre post0

GoboKernel: Routines with some specification

# routines

2000

1500

1000

500

0

pre post0

400

revision

600

800

100

200

300

Quickgraph: Routines with some specification

pre 200 post pre or post

Boogie: Routines with some specification

pre post0

20

40 revision

60

80

50 revision

100

Rxx: Routines with some specification

pre or post

pre post0

CCI: Routines with some specification

150

100

pre or post

pre or post

Fig. 33. Non-flat: # routines with specification (C# projects)

revision

50

0

# routines

250

200

150

200

100

12 10 8 6 4 2 0 1200 1000 800 600

# routines # routines

300 250 200 150

0

400

# routines # routines

pre or post

pre 0 post

150

250

revision

200

300

350

10

20

revision

30

40

50

shweet: Routines with some specification

pre 100 post pre or post

Dafny: Routines with some specification

60

400

250 200 150 100 50

100 50 0 10 8 6 4 2 0

55

# routines

pre or post

pre 0 post

10

20 revision

30

40

Labs: Routines with some specification

50

# routines

# routines

100

80

60

40

20

0

80

60

40

20

pre or post

pre 0 post

100

revision

200

300

5

15

revision

20

25

30

400

2000 pre or post

pre 0 post

200 revision

400

600

800

5

10 revision

15

20

25

RCC: Routines with some specification

pre or post

pre 0 post

ESCJava: Routines with some specification

Fig. 34. Non-flat: # routines with specification (Java projects)

10

Logging: Routines with some specification

pre or post

pre 0 post

DirectVCGen: Routines with some specification

# routines # routines

1500 1000 500 0 250 200 150 100 50

30

# routines # routines

800 600 400 200 6 5 4 3 2 1 0

56 pre or post

pre post0

100

200 revision

300

50 revision

100

Umbra: Routines with some specification

pre or post

pre post0

JavaFE: Routines with some specification

150

400

0.55

50

100

revision

150

200

250

300

inv

pre post0

100

150

revision

200

250

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: % routines and classes with specification

inv

pre post0

EiffelBase: % routines and classes with specification

Fig. 35. Non-flat: % routines and classes with specification (Eiffel projects)

50

GoboStructure: % routines and classes with specification

inv

pre post0

AutoTest: % routines and classes with specification

50 revision

100

150

200

inv

pre post0 50

revision

100

150

200

GoboUtility: % routines and classes with specification

inv

pre post0

EiffelProgramAnalysis: % routines and classes with specification 1.0 0.8

0.40 0.35 0.30

0.50

0.45

0.40

0.35

0.30

0.25

0.40

% routines or classes

% routines or classes

0.35

0.30

0.25

0.20

0.15

0.10

% routines or classes % routines or classes

0.25 0.20 0.15 0.6 0.5 0.4 0.3 0.2 0.1 0.0

% routines or classes % routines or classes

0.15 0.10 0.05 0.00 0.8 0.6 0.4 0.2 0.0

% routines or classes % routines or classes

0.6 0.4 0.2 0.0 0.4 0.3 0.2 0.1 0.0

57

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: % routines and classes with specification

inv

pre post0

GoboKernel: % routines and classes with specification

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.20

0.15

0.10

200

400

revision

600

800

pre post0

100

200

300

Quickgraph: % routines and classes with specification

pre post inv

Boogie: % routines and classes with specification

inv

inv

pre post0

inv

pre post0 40 revision

60

80

100

50 revision

100

150

Rxx: % routines and classes with specification

20

CCI: % routines and classes with specification

Fig. 36. Non-flat: % routines and classes with specification (C# projects)

revision

0.05

0.00

% routines or classes

% routines or classes

0.05 0.04

0.6 0.5 0.4 0.3 0.2

0.03 0.02 0.01 0.00 0.8 0.6 0.4

% routines or classes % routines or classes 0.2 0.0

% routines or classes % routines or classes

pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: % routines and classes with specification

pre post inv

Dafny: % routines and classes with specification 0.40 0.35 0.30 0.25 0.20 0.15 0.10

0.1 0.0 0.4 0.3 0.2 0.1 0.0

58

% routines or classes

inv

pre 0 post

10

20 revision

30

40

50

Labs: % routines and classes with specification

0.3

100

revision

200

300

400

inv

pre 0 post

10

15

revision

20

25

30

0.7 inv

pre 0 post

inv

pre 0 post revision

400

600

800

5

10 revision

15

20

25

30

RCC: % routines and classes with specification

200

ESCJava: % routines and classes with specification

Fig. 37. Non-flat: % routines and classes with specification (Java projects)

5

Logging: % routines and classes with specification

inv

pre 0 post

DirectVCGen: % routines and classes with specification 0.7 0.6 0.5

0.6 0.5 0.4 0.3

0.2

0.1

0.0

% routines or classes

% routines or classes

0.35

0.30

0.25

0.20

0.15

0.10

0.05

% routines or classes % routines or classes

0.2 0.1 0.5 0.4 0.3 0.2 0.1

% routines or classes % routines or classes

0.4 0.3 0.2 0.1 0.08 0.06 0.04 0.02 0.00

59 inv

100

200 revision

300

400

50 revision

100

150

Umbra: % routines and classes with specification

pre post0

inv

pre post0

JavaFE: % routines and classes with specification

400

50

100

revision

150

200

250

300

inv

pre post0

100

150

revision

200

250

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: Routines and classes with specification

inv

pre post0

EiffelBase: Routines and classes with specification

Fig. 38. Non-flat: # routines and classes with specification (Eiffel projects)

50

GoboStructure: Routines and classes with specification

inv

pre post0

AutoTest: Routines and classes with specification

800 600

50 revision

100

150

200

inv

pre post0 50

revision

100

150

200

GoboUtility: Routines and classes with specification

inv

pre post0

EiffelProgramAnalysis: Routines and classes with specification 300 250 200 150

300

200

100

# routines or classes

# routines or classes

120

100

80

60

40

20

# routines or classes # routines or classes

400 200 0 200 150 100 50 0

# routines or classes # routines or classes

150 100 50 0 120 100 80 60 40 20 0

# routines or classes # routines or classes

100 50 0 1000 800 600 400 200 0

60

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: Routines and classes with specification

inv

pre post0

GoboKernel: Routines and classes with specification

# routines or classes

1400

800 1000

600

400

200

0

250

200

150

100

200

400

revision

600

800

pre post0

100

200

300

Quickgraph: Routines and classes with specification

pre post inv

Boogie: Routines and classes with specification

pre post0

inv

pre post0 40 revision

60

80

50 revision

100

150

100

Rxx: Routines and classes with specification

20

CCI: Routines and classes with specification

inv

inv

Fig. 39. Non-flat: # routines and classes with specification (C# projects)

revision

50

0

# routines or classes

12 10 8 6 4 2 0 1000 800 600 200

400

# routines or classes # routines or classes

250 200

0

pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: Routines and classes with specification

pre post inv

Dafny: Routines and classes with specification

150 100 50

150 100 50 0 8 6 4

# routines or classes # routines or classes 2 0

61

# routines or classes

inv

pre 0 post

10

20 revision

30

40

50

Labs: Routines and classes with specification

100

80

60

40

20

0

# routines or classes

# routines or classes

60

50

40

30

20

10

0

inv

100

revision

200

300

400

10

15

revision

20

25

30

2000 1500 inv

pre 0 post

inv

pre 0 post revision

400

600

800

5

10 revision

15

20

25

30

RCC: Routines and classes with specification

200

ESCJava: Routines and classes with specification

Fig. 40. Non-flat: # routines and classes with specification (Java projects)

5

Logging: Routines and classes with specification

pre 0 post

inv

pre 0 post

DirectVCGen: Routines and classes with specification

# routines or classes # routines or classes

1000 500 0 200 150 100 50 0

# routines or classes # routines or classes

800 600 400 200 4 3 2 1 0

62 inv

100

200 revision

300

400

50 revision

100

150

Umbra: Routines and classes with specification

pre post0

inv

pre post0

JavaFE: Routines and classes with specification

50

100

revision

150

200

250

300

inv

pre post0

100

150

revision

200

250

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: Specification clauses per routine or class

inv

pre post0

EiffelBase: Specification clauses per routine or class

Fig. 41. Non-flat: Specification clauses per routine or class (Eiffel projects)

50

GoboStructure: Specification clauses per routine or class

inv

pre post0

AutoTest: Specification clauses per routine or class

0.8 0.6

50 revision

100

150

200

inv

pre post0 50

revision

100

150

200

GoboUtility: Specification clauses per routine or class

inv

pre post0

EiffelProgramAnalysis: Specification clauses per routine or class 0.20 0.15 0.10

1.0 0.8 0.6

1.5

1.0

0.5

# clauses per routine or class

# clauses per routine or class

1.2

1.0

0.8

0.6

0.4

0.2

# clauses per routine or class # clauses per routine or class

0.4 0.2 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

0.05 0.00 1.5 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

0.4 0.2 0.0 1.2 1.0 0.8 0.6 0.4 0.2 0.0

63

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: Specification clauses per routine or class

inv

pre post0

GoboKernel: Specification clauses per routine or class

3.0

2.5

200

400

revision

600

800

inv

pre post0

200

revision

300 inv

pre post0

inv

pre post0 40 revision

60

80

100

50 revision

100

150

Rxx: Specification clauses per routine or class

20

CCI: Specification clauses per routine or class

Fig. 42. Non-flat: Specification clauses per routine or class (C# projects)

100

Quickgraph: Specification clauses per routine or class

pre post inv

Boogie: Specification clauses per routine or class 0.06 0.05 0.04 0.03

2.0

1.5

1.0

0.5

# clauses per routine or class

# clauses per routine or class

0.0

0.3

0.2

0.1

0.0

# clauses per routine or class # clauses per routine or class

0.02 0.01 0.00 2.0 1.5 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: Specification clauses per routine or class

pre post inv

Dafny: Specification clauses per routine or class 1.0 0.8 0.6 0.4 0.2

1.5 1.0 0.5 0.0 0.6 0.5 0.4 0.3 0.2 0.1 0.0

64

# clauses per routine or class

inv

pre 0 post

10

20 revision

30

40

50

Labs: Specification clauses per routine or class

100

revision

200

300

400

inv

pre 0 post

10

15

revision

20

25

30 inv

pre 0 post

inv

pre 0 post revision

400

600

800

5

10 revision

15

20

25

30

RCC: Specification clauses per routine or class

200

ESCJava: Specification clauses per routine or class

Fig. 43. Non-flat: Specification clauses per routine or class (Java projects)

5

Logging: Specification clauses per routine or class

inv

pre 0 post

0.8 0.6

DirectVCGen: Specification clauses per routine or class 2.5 2.0 1.5

0.3

0.2

0.1

0.0

# clauses per routine or class

# clauses per routine or class

0.6

0.5

0.4

0.3

0.2

0.1

# clauses per routine or class # clauses per routine or class

0.4 0.2 0.5 0.4 0.3 0.2 0.1

# clauses per routine or class # clauses per routine or class

1.0 0.5 0.15 0.10 0.05 0.00

65 inv

100

200 revision

300

400

50 revision

100

150

Umbra: Specification clauses per routine or class

pre post0

inv

pre post0

JavaFE: Specification clauses per routine or class

inv

pre post0

50

100

revision

150

200

250

300

# clauses per routine or class

inv

pre post0 200

400

800

revision

600

1000

1200 inv

pre post0 50 revision

100

150

200 inv

pre post0

100

200

400 revision

300

500

600

EiffelBase: Average specification strength per routine or class EiffelProgramAnalysis: Average specification strength per routine or GoboKernel: class Average specification strength per routine or class

# clauses per routine or class

AutoTest: Average specification strength per routine or class

# clauses per routine or class

50

100

150

revision

200

250 inv

pre post0 20

40

60 revision

80

100

120

# clauses per routine or class

# clauses per routine or class

3.5

3.0

2.5

2.0

1.5

3.0 2.5 2.0 1.5 1.0

6 5 4 3 2 1 0

0.5 0.0

# clauses per routine or class

Fig. 44. Non-flat: Average specification strength per routine or class (Eiffel projects)

inv

pre post0 inv

pre post0 50

revision

100

150

200

# clauses per routine or class

inv

pre post0

200

revision

400

600

800

GoboStructure: Average specification strength per routine or classGoboTime: Average specification strength per routine or class GoboUtility: Average specification strength per routine or class GoboXML: Average specification strength per routine or class

# clauses per routine or class

3.5

3.0

2.5

2.0

1.5

2.6 2.4 2.2 2.0 1.8 1.6 1.4 1.2

1.2 1.0 0.8 0.6 0.4 0.2 0.0

3.0 2.5 2.0 1.5 1.0 0.5 0.0 2.5 2.0 1.5 1.0 0.5 0.0

66

# clauses per routine or class

200

400

revision

600

800

inv

100

200

revision

300

20

40 revision

60

80

100

inv

pre post0 50 revision

100

150

Rxx: Average specification strength per routine or class

inv

pre post0

Fig. 45. Non-flat: Average specification strength per routine or class (C# projects)

pre post0

Quickgraph: Average specification strength per routine or class

pre post inv

CCI: Average specification strength per routine or class 2.0

Boogie: Average specification strength per routine or class

5

4

100

150

250

revision

200

300

350

400

inv

pre 0 post 10

20

revision

30

40

50

60

shweet: Average specification strength per routine or class

pre post inv

Dafny: Average specification strength per routine or class 3.0 2.5 2.0 1.5 1.0

1.5 1.0 0.5

3

2

1

2.0

1.5

1.0

0.5

0.0

# clauses per routine or class

# clauses per routine or class # clauses per routine or class

0.0 8 6 4 2 0

# clauses per routine or class # clauses per routine or class

inv

pre 0 post

10

20 revision

30

40

50

Labs: Average specification strength per routine or class 2.4 2.2 2.0 1.8 1.6 1.4 1.2

0.5 0.0 2.0 1.5 1.0 0.5 0.0

67

# clauses per routine or class

100

revision

200

300

400

inv

5

10

15

revision

20

25

30

200 revision

400

600

800

inv

pre 0 post 5

10 revision

15

20

25

30

RCC: Average specification strength per routine or class

inv

pre 0 post

Fig. 46. Non-flat: Average specification strength per routine or class (Java projects)

pre 0 post

Logging: Average specification strength per routine or class

inv

pre 0 post

DirectVCGen: Average specification strength per routine or class ESCJava: Average specification strength per routine or class

# clauses per routine or class

2.0

1.5

1.0

0.5

0.0

2.0

100

200 revision

300

400

inv

pre post0

50 revision

100

150

Umbra: Average specification strength per routine or class

inv

pre post0

JavaFE: Average specification strength per routine or class 5 4

3.5 3.0 2.5 2.0 1.5

1.8

1.6

1.4

1.2

1.0

# clauses per routine or class

# clauses per routine or class # clauses per routine or class

1.0 6 5 4 3 2 1

# clauses per routine or class # clauses per routine or class

3 2 2.0 1.5 1.0 0.5 0.0

68

# routines

# routines

6

5

4

3

2

1

0

6

100

150

revision

200

250

300

inv strong

pre weak pre strong post weak post strong 0 inv weak

50

150

revision

200

250

400

800

revision

600

1000

1200

pre weak pre strong post weak post strong 20 inv 0 weak inv strong 40 revision

60

80

100

GoboTime: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 200 inv weak

EiffelBase: Weakening and strengthening

Fig. 47. Non-flat: Weakening and strengthening (Eiffel projects)

100

GoboStructure: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 50 inv weak

AutoTest: Weakening and strengthening 30 25 20 15 10 5 0 2.0 1.5 1.0

5

4

3

2

1

0

# routines # routines

120

3.0 2.5 2.0 1.5 1.0 0.5 0.0

50

100 revision

150

200

pre weak pre strong post weak post strong 0 inv weak 50

100

150

200

GoboUtility: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 inv weak

EiffelProgramAnalysis: Weakening and strengthening

inv strong

revision

6 5 4 3 2 1

# routines # routines

8 6 4 2 0 14

0

0.5 0.0

# routines # routines

12 10 8 6 4 2 0

69

200

400 revision

300

500

600

inv strong

pre weak pre strong post weak post strong 0 inv weak

200

revision

400

600

800

GoboXML: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 100 inv weak

GoboKernel: Weakening and strengthening

# routines

# routines

400

revision

600

800

inv strong

pre weak pre strong post weak post strong 0 inv weak

100

300

pre weak pre strong post weak post strong inv 0 weak inv strong

pre weak pre strong post weak post strong inv 0weak inv strong 40 revision

60

80

50 revision

100

Rxx: Weakening and strengthening

20

CCI: Weakening and strengthening

Fig. 48. Non-flat: Weakening and strengthening (C# projects)

revision

200

Quickgraph: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong inv weak 200

Boogie: Weakening and strengthening

150

100

8 6

3.0 2.5 2.0 1.5

250

200

150

100

50

0

120

100

80

60

40

20

0

# routines # routines

1.0 0.5 0.0 12 10 8 6 4 2 0

# routines # routines

150

250

revision

200

300

350

20

revision

30

40

50

shweet: Weakening and strengthening

pre weak pre strong post weak post strong 10 inv0weak inv strong

inv strong

pre weak pre strong post weak post strong inv weak 100

Dafny: Weakening and strengthening

60

400

40 30 20 10 0

4 2 0 2.0 1.5 1.0 0.5 0.0

70

# routines

pre weak pre strong post weak post strong inv0weak inv strong 10

20 revision

30

40

Labs: Weakening and strengthening

50

# routines

# routines

100

200

revision

300

pre weak pre strong post weak post strong 5 inv0 weak inv strong

15

revision

20

25

30

400

200 revision

400

600

800

10 revision

15

20

25

RCC: Weakening and strengthening

pre weak pre strong post weak post strong inv0 weak 5 inv strong

pre weak pre strong post weak post strong inv 0weak inv strong

ESCJava: Weakening and strengthening

Fig. 49. Non-flat: Weakening and strengthening (Java projects)

10

Logging: Weakening and strengthening

pre weak pre strong post weak post strong inv0 weak inv strong

DirectVCGen: Weakening and strengthening 300 250 200

30

70 60 50

10

8

6

4

2

0

20

15

10

5

0

# routines # routines

150 100 50 0 200 150 100 50 0

# routines # routines

40 30 20 10 0 1.0 0.8 0.6 0.4 0.2 0.0

71

100

200 revision

300

50 revision

100

Umbra: Weakening and strengthening

pre weak pre strong post weak post strong inv 0 weak inv strong

inv strong

pre weak pre strong post weak post strong 0 inv weak

JavaFE: Weakening and strengthening

150

400

% routines

% routines

0.60

0.55

0.50

0.45

0.40

0.8

0.7

0.6

0.5

0.4

0.3

0.2

100

150

revision

200

250

300

50

100

150

revision

200

250

400

800

revision

600

1000

1200

0 20 public non−public

40

60

80

100

GoboTime: % routines with pre or post

0 200 public non−public

EiffelBase: % routines with pre or post

revision

120

Fig. 50. Non-flat: % routines with specification: public vs nonpublic (Eiffel projects)

0 public non−public

GoboStructure: % routines with pre or post

0 50 public non−public

AutoTest: % routines with pre or post

0.4 0.3 0.2 0.1 1.0 0.8 0.6 0.2

% routines % routines

0.30 0.25 0.20

0.0

0.4

% routines % routines

0.15 0.10 0.05 0.00 1.0 0.8 0.6 0.4 0.2 0.0

50

100 revision

150

0 public non−public

50

revision

100

150

200

200

GoboUtility: % routines with pre or post

0 public non−public

EiffelProgramAnalysis: % routines with pre or post

% routines % routines

1.0 0.8 0.6 0.4 0.2 0.0 0.6 0.5 0.4 0.3 0.2 0.1 0.0

72

200

400 revision

300

500

600

0 public non−public

200

revision

400

600

800

GoboXML: % routines with pre or post

0 100 public non−public

GoboKernel: % routines with pre or post

% routines

% routines

0.80

0.75

0.70

0.65

0.60

0.55

0.50

0.45

0.20

0.15

400

revision

600

800

0 public non−public

100

200

300

Quickgraph: % routines with pre or post

200 public non−public

Boogie: % routines with pre or post 0.12 0 public non−public

0 public non−public

40 revision

60

80

50 revision

100

Rxx: % routines with pre or post

20

CCI: % routines with pre or post

150

100

Fig. 51. Non-flat: % routines with specification: public vs nonpublic (C# projects)

revision

0.05

0.7 0.6 0.5

0.10 0.08

0.00

0.10

% routines % routines

0.06 0.04 0.02 0.00 1.0 0.8 0.6 0.4 0.2 0.0

% routines % routines

150

250

revision

200

300

350

0 10 public non−public

20

revision

30

40

50

shweet: % routines with pre or post

public 100 non−public

Dafny: % routines with pre or post

60

400

0.50 0.45 0.40 0.35

0.4 0.3 0.2 0.4 0.3 0.2 0.1 0.0

73

% routines

0 public non−public

10

20 revision

30

40

Labs: % routines with pre or post

50

% routines

% routines

100

200

revision

300

15

revision

20

25

30

400

0 5 public non−public

0 public non−public revision

400

600

10 revision

15

20

25

RCC: % routines with pre or post

200

800

ESCJava: % routines with pre or post

30

Fig. 52. Non-flat: % routines with specification: public vs nonpublic (Java projects)

10

Logging: % routines with pre or post

0 5 public non−public

0 public non−public

DirectVCGen: % routines with pre or post 0.7 0.6 0.5

0.3

0.2

0.1

0.0

0.5

0.4

0.3

0.2

0.1

% routines % routines

0.4 0.3 0.2 0.6 0.5 0.4 0.3 0.2 0.1

% routines % routines

0.8 0.7 0.6 0.5 0.4 0.25 0.20 0.15 0.10 0.05 0.00

74 0 public non−public

0 public non−public

200 revision

300

50 revision

100

Umbra: % routines with pre or post

100

JavaFE: % routines with pre or post

150

400

15

10

5

0

# classes

−A weak

+A same +A strong +A weak −A same 0 −A strong

50

100

150

revision

200

250

300 −A weak

+A same +A strong +A weak −A same 0 200 −A strong 400

800

revision

600

1000

1200

50

100

150

revision

200

250

+A same +A strong +A weak −A same −A 0 strong −A weak 20

40

60 revision

80

100

120

Fig. 53. Non-flat: Weakening and strengthening of class invariants (Eiffel projects)

−A weak

+A same +A strong +A weak −A same 0 −A strong

50

100 revision

150

200

−A weak

+A same +A strong +A weak −A same 0 −A strong 50

revision

100

150

200

GoboUtility: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 −A strong

200

400 revision

300

500

600

−A weak

+A same +A strong +A weak −A same 0 −A strong

200

revision

400

600

800

GoboXML: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 100 −A strong

EiffelBase: Weakening and strengthening of class invariants EiffelProgramAnalysis: Weakening and strengthening of class invariants GoboKernel: Weakening and strengthening of class invariants

GoboStructure: Weakening and strengthening of class invariants GoboTime: Weakening and strengthening of class invariants

# classes

15

10

5

0

14 12 10 8 6 4 2 0

3.0 2.5

AutoTest: Weakening and strengthening of class invariants

# classes # classes

3.0 2.5 2.0 1.5 1.0 0.5 0.0

2.0 1.5 1.0 0.5

2.0 1.5 1.0 0.5 0.0 1.0 0.8 0.6 0.4 0.2 0.0

# classes # classes

# classes # classes

0.0 12 10 8 6 4 2 0

75

# classes

400

revision

600

800

100

200

revision

300

1.0

20

40 revision

60

80

100

+A same +A strong +A weak −A same −A 0 strong −A weak 50 revision

100

150

Rxx: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A 0strong −A weak

CCI: Weakening and strengthening of class invariants

Fig. 54. Non-flat: Weakening and strengthening of class invariants (C# projects)

−A weak

+A same +A strong +A weak −A same 0 −A strong

Quickgraph: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same −A strong200

Boogie: Weakening and strengthening of class invariants

1.0

0.5

150

250

revision

200

300

350

400

+A same +A strong +A weak −A same −A0strong −A weak 10

20

revision

30

40

50

60

shweet: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same −A strong100

Dafny: Weakening and strengthening of class invariants 1.0 0.5 0.0

0.5 0.0 −0.5

0.0

−0.5

−1.0

1.0

0.5

0.0

−0.5

−1.0

# classes

# classes # classes

−1.0 1.0 0.5 0.0 −0.5 −1.0

# classes # classes

1.0 0.5 0.0 −0.5 −1.0

−0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

76

# classes

+A same +A strong +A weak −A same −A0strong −A weak 10

20 revision

30

40

50

Labs: Weakening and strengthening of class invariants

# classes

# classes

100

200

revision

300

400

5

10

15

revision

20

25

30

200 revision

400

600

800

+A same +A strong +A weak −A same −A0 strong −A weak 5

10 revision

15

20

25

30

RCC: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A 0strong −A weak

ESCJava: Weakening and strengthening of class invariants

Fig. 55. Non-flat: Weakening and strengthening of class invariants (Java projects)

+A same +A strong +A weak −A same −A0 strong −A weak

Logging: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A0 strong −A weak

DirectVCGen: Weakening and strengthening of class invariants 1.0 0.5

100

200 revision

300

400

−A weak

+A same +A strong +A weak −A same 0 −A strong 50

revision

100

150

Umbra: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 −A strong

JavaFE: Weakening and strengthening of class invariants 1.0 0.5

1.0

0.5

0.0

−0.5

−1.0

1.0

0.5

0.0

−0.5

−1.0

# classes # classes

0.0 −0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

# classes # classes

0.0 −0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

77

12

Flat classes

The following pages show the plots for the 21 projects of our study, considering flat classes.

78

# routines

# routines

25

revision

150

200

250

300

spec

body+spec 0 (nonempty 50 spec) 100 body revision

150

200

300

800

revision

600

1000

1200

spec

body+spec 0 (nonempty 20 spec) 40 body revision

60

80

100

120

GoboTime: # changed routines (for nonempty spec)

spec

body+spec 200 spec) 400 body0 (nonempty

EiffelBase: # changed routines (for nonempty spec)

revision

100

150

200

body+spec 0 (nonempty 50 body spec) spec

revision

100

150

200

GoboUtility: # changed routines (for nonempty spec)

body+spec 0 (nonempty spec) 50 body spec

EiffelProgramAnalysis: # changed routines (for nonempty spec)

Fig. 56. Flat: Changed routines (with nonempty unchanged spec) (Eiffel projects)

250

GoboStructure: # changed routines (for nonempty spec)

spec

body+spec 0 (nonempty 50 spec)100 body

AutoTest: # changed routines (for nonempty spec) 70 60 50 40

35 30 25 20

20

15

10

5

0

200

150

100

50

0

# routines # routines

200 100 0 20 15 10 5 0

# routines # routines

30 20 10 0 12 10 8 6 4 2 0

# routines # routines

15 10 5 0 1500 1000 500 0

79

400 revision

300

500

600

body+spec body0 (nonempty200 spec) spec

revision

400

600

800

GoboXML: # changed routines (for nonempty spec)

spec

body+spec 0 (nonempty 100 spec) 200 body

GoboKernel: # changed routines (for nonempty spec)

# routines

200

150

100

50

0

120

100

80

400 revision

600

800

body+spec 0 (nonempty spec) 100 body spec 200

300

Quickgraph: # changed routines (for nonempty spec)

spec

body+spec 200 body (nonempty spec)

Boogie: # changed routines (for nonempty spec)

revision

40 revision

60

80

body+spec 0 (nonempty spec) 50 body spec revision

100

150

100

Rxx: # changed routines (for nonempty spec)

body+spec 0 (nonempty 20spec) body spec

CCI: # changed routines (for nonempty spec)

250

revision

200

300

350

400

spec

body+spec 0 (nonempty 10 spec) 20 body

revision

30

40

50

60

shweet: # changed routines (for nonempty spec)

spec

body+spec 100 150 body (nonempty spec)

Dafny: # changed routines (for nonempty spec)

Fig. 57. Flat: Changed routines (with nonempty unchanged spec) (C# projects)

40

20

# routines

2.0 1.5

70 60 50 40 30 20

1.0 0.5 0.0 150 100

0

60

# routines # routines 50 0

# routines # routines

140 120 100 80 60 40 20 0

10 0 2.0 1.5 1.0 0.5 0.0

80

# routines

spec

body+spec 0 (nonempty 10 spec) body

20 revision

30

40

50

Labs: # changed routines (for nonempty spec)

20 15 10

# routines

30 20 10

200 revision

300

400

Logging: # changed routines (for nonempty spec)

body+spec 0 (nonempty 100 body spec) spec

body+spec 0 (nonempty 5 spec)10 body 15

20

25

30

30 25

0

300 spec

body+spec 0 (nonempty 5 body spec)10 spec

revision

400

600

800

revision

15

20

25

30

RCC: # changed routines (for nonempty spec)

body+spec 0 (nonempty200 body spec) spec

ESCJava: # changed routines (for nonempty spec)

200 revision

300

400

body+spec 0 (nonempty spec)50 body spec

revision

100

150

Umbra: # changed routines (for nonempty spec)

body+spec 0 (nonempty spec) 100 body spec

JavaFE: # changed routines (for nonempty spec)

Fig. 58. Flat: Changed routines (with nonempty unchanged spec) (Java projects)

revision

5

300 250 200

250 200 150 100 50 0 600

0

# routines

DirectVCGen: # changed routines (for nonempty spec)

# routines # routines

500 400 300 200 100 0

# routines # routines

150 100 50 0 20 15 10 5 0

81

# routines

# routines

25

100

150 revision

200

250

300

0 body+spec spec

50

100 revision

150

250

300

400

800

revision

600

1000

1200

0 20 body+spec spec

40 revision

60

80

100

120

GoboTime: # changed routines (w/ or w/o body change)

0 200 body+spec spec

EiffelBase: # changed routines (w/ or w/o body change)

50

100 revision

150

200

0 body+spec spec

50

revision

100

150

200

GoboUtility: # changed routines (w/ or w/o body change)

0 body+spec spec

200

400 revision

300

500

600

0 body+spec spec

200

revision

400

600

800

GoboXML: # changed routines (w/ or w/o body change)

0 100 body+spec spec

EiffelProgramAnalysis: # changed routines (w/ or w/o body change) GoboKernel: # changed routines (w/ or w/o body change)

Fig. 59. Flat: Routines with changed specification (with or without body changing) (Eiffel projects)

200

GoboStructure: # changed routines (w/ or w/o body change)

0 50 body+spec spec

AutoTest: # changed routines (w/ or w/o body change) 70 60 50 40

35 30 25 20

20

15

10

5

0

200

150

100

50

0

# routines # routines

200 100 0 14 12 10 8 6 4 2 0

# routines # routines

30 20 10 0 10 8 6 4 2 0

# routines # routines

15 10 5 0 1500 1000 500 0

82

# routines

# routines

200

150

100

50

200 body+spec spec

400 revision

600

800

Boogie: # changed routines (w/ or w/o body change)

0 body+spec spec

100 revision

200

2.0 1.5

20

40 revision

60

80

100

0 body+spec spec

50 revision

100

150

Rxx: # changed routines (w/ or w/o body change)

0 body+spec spec

CCI: # changed routines (w/ or w/o body change)

150

250

revision

200

300

350

400

0 10 body+spec spec

20

revision

30

40

50

60

shweet: # changed routines (w/ or w/o body change)

100 body+spec spec

Dafny: # changed routines (w/ or w/o body change)

Fig. 60. Flat: Routines with changed specification (with or without body changing) (C# projects)

300

Quickgraph: # changed routines (w/ or w/o body change)

0

120

100

80

60

40

20

0

# routines # routines

70 60 50 40 30 20

1.0 0.5 0.0 15 10 5 0

# routines # routines

140 120 100 80 60 40 20 0

10 0 2.0 1.5 1.0 0.5 0.0

83

# routines

0 body+spec spec

10

20 revision

30

40

50

Labs: # changed routines (w/ or w/o body change)

# routines

8 6 4 2 0

100

200 revision

300

400

Logging: # changed routines (w/ or w/o body change)

0 body+spec spec

0 5 body+spec spec

10

15

20

25

30

30 25 20 15 10

200 revision

400

600

800

0 5 body+spec spec

10

revision

15

20

25

30

RCC: # changed routines (w/ or w/o body change)

0 body+spec spec

ESCJava: # changed routines (w/ or w/o body change)

100

200 revision

300

400

0 body+spec spec

50 revision

100

150

Umbra: # changed routines (w/ or w/o body change)

0 body+spec spec

JavaFE: # changed routines (w/ or w/o body change)

Fig. 61. Flat: Routines with changed specification (with or without body changing) (Java projects)

revision

5 0

# routines

300 250 200 150 100 50 0 600

300 250 200

DirectVCGen: # changed routines (w/ or w/o body change)

# routines # routines

500 400 300 200 100 0

# routines # routines

150 100 50 0 20 15 10 5 0

84

# routines

# routines

80

60

40

20

0

revision

150

200

250

300

spec

body+spec body 0 (nonempty 50 spec) 100 body revision

150

200

250

GoboStructure: # changed routines (types of change)

spec

body+spec body 0 (nonempty 50 spec)100 body

AutoTest: # changed routines (types of change) 700 600 500 400 300 200 100 0 30 25 20

400

300

200

100

0

# routines

800

revision

600

1000

1200

spec

body+spec body 0 (nonempty 20 spec) 40 body revision

60

80

100

120

GoboTime: # changed routines (types of change)

spec

body+spec body 200 spec) 400 body0 (nonempty

EiffelBase: # changed routines (types of change)

revision

100

150

200

body+spec body 0 (nonempty 50 body spec) spec

100

150

200

GoboUtility: # changed routines (types of change)

body+spec body 0 (nonempty spec) 50 body spec

EiffelProgramAnalysis: # changed routines (types of change)

revision

Fig. 62. Flat: changed routines (types of change) (Eiffel projects)

# routines

250 200 150 100 50 0 20 15 10 5

# routines # routines

50 40 30 20 10 0 1500 1000 0

15 10 5 0

# routines # routines

500 0

85

400 revision

300

500

600

body+spec body body0 (nonempty200 spec) spec

revision

400

600

800

GoboXML: # changed routines (types of change)

spec

body+spec body 0 (nonempty 100 spec) 200 body

GoboKernel: # changed routines (types of change)

# routines

# routines

400 revision

600

800

body+spec body 0 (nonempty spec) 100 body spec revision

200

300

Quickgraph: # changed routines (types of change)

spec

body+spec body 200 body (nonempty spec)

Boogie: # changed routines (types of change) 10 8 6

200

150

100

50

0

120

100

80

60

40

20

0

# routines

40 revision

60

80

body+spec body 0 (nonempty spec) 50 body spec revision

100

Rxx: # changed routines (types of change)

body+spec body 0 (nonempty 20spec) body spec

CCI: # changed routines (types of change)

150

100

spec

250

revision

200

300

350

revision

30

40

50

shweet: # changed routines (types of change)

body+spec body 0 (nonempty 10 spec) 20 body

spec

body+spec body 100 150 body (nonempty spec)

Dafny: # changed routines (types of change)

Fig. 63. Flat: changed routines (types of change) (C# projects)

# routines

70 60 50 40 30 20

4 2 0 150 100 50 0

# routines # routines

60

400

140 120 100 80 60 40 20 0

10 0 25 20 15 10 5 0

86

# routines

spec

body+spec body 0 (nonempty 10 spec) body

20 revision

30

40

Labs: # changed routines (types of change)

50

100 20 15 10

300

Logging: # changed routines (types of change)

revision

200

30

400 revision

400

600

800

body+spec body 0 (nonempty 5 spec)10 body 15

20

25

body+spec body 0 (nonempty 5 body spec)10

15

20

25

RCC: # changed routines (types of change)

body+spec body 0 (nonempty200 body spec) spec

30

ESCJava: # changed routines (types of change)

spec revision

revision

200 revision

300

body+spec body 0 (nonempty spec)50 body spec

revision

100

400

150

Umbra: # changed routines (types of change)

body+spec body 0 (nonempty spec) 100 body spec

JavaFE: # changed routines (types of change)

Fig. 64. Flat: changed routines (types of change) (Java projects)

spec

5 0

# routines

80 60 40 20 30 25

0

300 250 200 150 100 50 0 600 500 400 300 200 100 0

# routines

body+spec body 0 (nonempty 100 body spec) spec

# routines # routines

300 250 200 150 100 50 140

0

DirectVCGen: # changed routines (types of change)

# routines # routines

120 100 80 60 40 20 0

87

4

3

2

1

0

20

15

10

5

0

100 revision

150

200

250

% inv mean inv # inv0 weak 50 # inv strong 100 revision

150

200

300

400

800

revision

600

1000

40 revision

60

80

1200

100

GoboTime: Average and % inv

% inv mean inv 0 weak 20 # inv # inv strong

% inv mean inv # inv0 weak 200 # inv strong

EiffelBase: Average and % inv

120

% inv mean inv 0 weak # inv # inv strong

% inv mean inv 0 weak # inv # inv strong revision

100

150

50

revision

100

150

GoboUtility: Average and % inv

50

200

200

EiffelProgramAnalysis: Average and % inv

Fig. 65. Flat: Average invariant strength and % of classes with invariant (Eiffel projects)

250

GoboStructure: Average and % inv

% inv mean inv # inv0 weak 50 # inv strong

AutoTest: Average and % inv 40 30 20 10 0 5 4 3 2 1 0

1.2 1.0 0.8 0.6 0.4 0.2 0.0 3.0 2.5 2.0 1.5 1.0 0.5 0.0

12 10 8 6 4 2 0 150 100 50 0

88

% inv mean inv # inv0 weak # inv strong

% inv mean inv # inv0 weak 100 # inv strong

400 revision

300

500

200

revision

400

600

GoboXML: Average and % inv

200

600

800

GoboKernel: Average and % inv

35

30

25

20

15

10

5

0

2.0

1.5

% inv mean inv # inv0 weak # inv strong

# inv strong

% inv mean inv 200 # inv weak revision

600

100

200

300

Quickgraph: Average and % inv

400

Boogie: Average and % inv

800

revision

% inv mean inv 0 weak # inv # inv strong

% inv mean inv 0 weak # inv # inv strong 20 revision

60

50 revision

100

Rxx: Average and % inv

40

CCI: Average and % inv

80

150

100

% inv mean inv 0 weak 10 # inv # inv strong

# inv strong

% inv mean inv 100 # inv weak

250

revision

200

300

20

revision

30

40

shweet: Average and % inv

150

Dafny: Average and % inv

Fig. 66. Flat: Average invariant strength and % of classes with invariant (C# projects)

0.5

0.0

1.0

2.0 1.5 1.0 0.5 0.0 8 6 4 2 0

8 6

50

350

60

400

8 6 4 2 0

4 2 0 1.0 0.5 0.0 −0.5 −1.0

89

% inv mean inv 0 weak # inv # inv strong

10

20 revision

30

Labs: Average and % inv

40

50

2.0 1.5 1.0 0.5 0.0 2.0 1.5 1.0 0.5 0.0 % inv mean inv # 0inv weak 5 # inv strong

% inv mean inv #0 inv weak # inv strong revision

200

300

10

20

25

30

400

% inv mean inv #0 inv weak 5 # inv strong

% inv mean inv 0 weak # inv # inv strong revision

400

600

10

15

20

RCC: Average and % inv

200

ESCJava: Average and % inv

25

800

30 revision

% inv mean inv 0 weak # inv # inv strong

% inv mean inv # inv0 weak # inv strong

200 revision

300

50 revision

100

Umbra: Average and % inv

100

JavaFE: Average and % inv

Fig. 67. Flat: Average invariant strength and % of classes with invariant (Java projects)

revision

15

Logging: Average and % inv

100

DirectVCGen: Average and % inv

5 4 3 2 1 0 6 5 4 3 1 0

2

6 5 4 3 2 1 0 6 5 4 3 2 1 0

90

150

400

400

300

200

100

# classes or invariants

# classes or invariants

800

600

400

200

0

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv 100 revision

150

200

250

50

100 revision

150

200

300

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv 400

800

revision

600

1000

1200

20

40 revision

60

80

100

GoboTime: # classes and invariants

200

EiffelBase: # classes and invariants

120

classes 0 w/ inv

# inv

classes 0 w/ inv

revision

100

150

50

100

150

GoboUtility: # classes and invariants

50

200

200

EiffelProgramAnalysis: # classes and invariants

# inv

revision

Fig. 68. Flat: # of classes, classes with invariants, invariant clauses (Eiffel projects)

250

GoboStructure: # classes and invariants

50

AutoTest: # classes and invariants 1400 1200 1000 800 600 400 200 0 80 60

# classes or invariants # classes or invariants

200 150 100 50 0 60 50 40 30 20 10 0

# classes or invariants # classes or invariants

150 100 50 0 6000 5000 4000 3000 2000

40 20 0

# classes or invariants # classes or invariants

1000 0

91

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv

200

400 revision

300

500

600

200

revision

400

600

800

GoboXML: # classes and invariants

100

GoboKernel: # classes and invariants

# inv

classes 0 w/ inv

400 revision

600

800

100 revision

200

300 # inv

classes 0 w/ inv

# inv

classes 0 w/ inv 40 revision

60

80

50 revision

100

Rxx: # classes and invariants

20

CCI: # classes and invariants

150

100

classes w/ 0 inv # inv

150

250

revision

200

300

20

revision

30

40

350

50

shweet: # classes and invariants

10

classes w/ inv 100 # inv

Dafny: # classes and invariants

Fig. 69. Flat: # of classes, classes with invariants, invariant clauses (C# projects)

Quickgraph: # classes and invariants

classes 200 w/ inv # inv

Boogie: # classes and invariants 100 80 60 40

300 250 200 150

800

600

400

200

0

# classes or invariants

# classes or invariants

300

250

200

150

100

50

0

# classes or invariants # classes or invariants

20 0 250 200 150 100 50 0

# classes or invariants # classes or invariants

60

400

100 80 60 40 20

100 50 0 35 30 25 20 15 10 5 0

92

# classes or invariants

classes w/ 0inv # inv

10

20 revision

30

40

Labs: # classes and invariants

50

# classes or invariants # classes or invariants

classes w/0inv # inv

classes w/0inv # inv 200 revision

300

5

10

20

25

30

400

classes w/0inv # inv

# inv

classes 0 w/ inv

5

revision

400

600

10

revision

15

20

25

RCC: # classes and invariants

200

ESCJava: # classes and invariants

800

30

# inv

classes 0 w/ inv

# inv

classes 0 w/ inv

200 revision

300

50 revision

100

Umbra: # classes and invariants

100

JavaFE: # classes and invariants

Fig. 70. Flat: # of classes, classes with invariants, invariant clauses (Java projects)

revision

15

Logging: # classes and invariants

100

DirectVCGen: # classes and invariants 300 250 200 150

500 400 300

80 60 40 20 0 20 15 10 5

# classes or invariants # classes or invariants

100 50 80 60 40 20

# classes or invariants # classes or invariants

200 100 80 60 40 20 0

93

150

400

# routines

# routines

50

100 revision

150

200

250

300

pre or post

pre post0 50

100 revision

150

200

250

GoboStructure: Routines with some specification

pre or post

pre post0

AutoTest: Routines with some specification 5000 4000 3000

1500

1000

500

2500

2000

1500

1000

500

# routines # routines

2000 1000 500 400 300 200 100 pre or post

pre post0

200

400

800

revision

600

1000

1200

20

40 revision

60

80

100

120

GoboTime: Routines with some specification

pre or post

pre post0

EiffelBase: Routines with some specification

pre post0

50

revision

100

150

200

50

100

150

200

GoboUtility: Routines with some specification

pre or post

pre post0

EiffelProgramAnalysis: Routines with some specification

pre or post

Fig. 71. Flat: # routines with specification (Eiffel projects)

0

2000 1500 1000 500 0 150 100

revision

50 0

# routines # routines

1000 800 600 400 200 0 60000 40000

# routines # routines

20000 0

94

pre or post

pre post0

100

200

400 revision

300

500

600

200

revision

400

600

800

GoboXML: Routines with some specification

pre or post

pre post0

GoboKernel: Routines with some specification

# routines

# routines

5000

4000

pre or post

pre post0

400 revision

600

800

100 revision

200

300

Quickgraph: Routines with some specification

pre 200 post pre or post

Boogie: Routines with some specification 12 10 8 6

40 revision

60

80

50 revision

100

150

100

pre or post

pre 0 post

150

250

revision

200

300

350

10

20

revision

30

40

50

shweet: Routines with some specification

pre 100 post pre or post

Dafny: Routines with some specification

Fig. 72. Flat: # routines with specification (C# projects)

pre or post

pre post0

20

Rxx: Routines with some specification

pre or post

pre post0

CCI: Routines with some specification 600 500 400 300

3000

2000

1000

0

250

200

150

100

50

0

# routines # routines

4 2 0 2000 1500 1000 500 0

# routines # routines

60

400

400 300 200 100

200 100 0 10 8 6 4 2 0

95

# routines

pre or post

pre 0 post

10

20 revision

30

40

Labs: Routines with some specification

50

# routines # routines

pre or post

pre 0 post

100

200 revision

300

5

10 revision

15

20

25

30

400

pre or post

pre 0 post

200 revision

400

600

800

5

10

revision

15

20

25

RCC: Routines with some specification

pre or post

pre 0 post

ESCJava: Routines with some specification

30

1000 pre or post

pre post0

200 revision

300

50 revision

100

Umbra: Routines with some specification

100

JavaFE: Routines with some specification

pre or post

pre post0

Fig. 73. Flat: # routines with specification (Java projects)

Logging: Routines with some specification

pre or post

pre 0 post

DirectVCGen: Routines with some specification 2500 2000

150 100 50 0 80 60 40 20

# routines # routines

1500 1000 500 0 600 500 400 300 200 100 0

# routines # routines

800 600 400 200 30 20 10 0

96

150

400

% routines or classes

% routines or classes

50

100 revision

150

200

250

300

inv

pre post0 50

100 revision

150

200

250

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: % routines and classes with specification

inv

pre post0

EiffelBase: % routines and classes with specification

50

revision

100

150

200

inv

pre post0

50

revision

100

150

200

GoboUtility: % routines and classes with specification

inv

pre post0

EiffelProgramAnalysis: % routines and classes with specification

0.15

Fig. 74. Flat: % routines and classes with specification (Eiffel projects)

GoboStructure: % routines and classes with specification

inv

pre post0

AutoTest: % routines and classes with specification 0.7 0.6

1.0 0.8 0.6

0.60

0.55

0.50

0.45

0.40

0.8

0.7

0.6

0.5

% routines or classes % routines or classes

0.5 0.4 0.3 0.6 0.4 0.2 0.0

% routines or classes % routines or classes

0.10 0.05 0.00 0.8 0.6 0.4 0.2 0.0

% routines or classes % routines or classes

0.4 0.2 0.0 0.8 0.6 0.4 0.2 0.0

97

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: % routines and classes with specification

inv

pre post0

GoboKernel: % routines and classes with specification

0.7

0.6

0.5

0.4

0.3

0.2

0.1

% routines or classes

% routines or classes

0.20

0.15

0.10

200

400 revision

600

800

pre post0 100

200

300

Quickgraph: % routines and classes with specification

pre post inv

Boogie: % routines and classes with specification

inv revision

inv

pre post0

inv

pre post0 40 revision

60

80

100

50 revision

100

150

Rxx: % routines and classes with specification

20

CCI: % routines and classes with specification

pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: % routines and classes with specification

pre post inv

Dafny: % routines and classes with specification

Fig. 75. Flat: % routines and classes with specification (C# projects)

0.05

0.00

0.5 0.4 0.3

0.05 0.04 0.03 0.02 0.01 0.00 0.8 0.6 0.4

% routines or classes % routines or classes 0.2 0.0

% routines or classes % routines or classes

0.40 0.35 0.30 0.25 0.20 0.15

0.2 0.1 0.0 0.4 0.3 0.2 0.1 0.0

98

% routines or classes

inv

pre 0 post

10

20 revision

30

40

50

Labs: % routines and classes with specification

0.15

100

200 revision

300

400

inv

pre 0 post 5

10 revision

15

20

30

0.7 inv

pre 0 post

inv

pre 0 post revision

400

600

800

5

10

revision

15

20

25

30

RCC: % routines and classes with specification

200

ESCJava: % routines and classes with specification

inv

100

200 revision

300

400

50 revision

100

150

Umbra: % routines and classes with specification

pre post0

inv

pre post0

JavaFE: % routines and classes with specification

Fig. 76. Flat: % routines and classes with specification (Java projects)

25

Logging: % routines and classes with specification

inv

pre 0 post

DirectVCGen: % routines and classes with specification

0.6 0.5

0.6 0.5 0.4 0.3

0.10 0.05 0.00

% routines or classes % routines or classes

0.35 0.30 0.25 0.20 0.15 0.10 0.05

% routines or classes % routines or classes

0.2 0.1 0.6 0.5 0.4 0.3 0.2 0.1 0.0

% routines or classes % routines or classes

0.4 0.3 0.2 0.1 0.08 0.06 0.04 0.02 0.00

99

# routines or classes

# routines or classes

50

100 revision

150

200

250

300

inv

pre post0 50

100 revision

150

200

250

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: Routines and classes with specification

inv

pre post0

EiffelBase: Routines and classes with specification

50

revision

100

150

200

inv

pre post0

50

revision

100

150

200

GoboUtility: Routines and classes with specification

inv

pre post0

800 600

EiffelProgramAnalysis: Routines and classes with specification 1500

Fig. 77. Flat: # routines and classes with specification (Eiffel projects)

GoboStructure: Routines and classes with specification

inv

pre post0

AutoTest: Routines and classes with specification

3000 2000

1000

800

600

400

200

1500

1000

500

0

# routines or classes # routines or classes

1000 500 0 400 300 200 100 0

# routines or classes # routines or classes

1000 500 0 150 100 50 0

# routines or classes # routines or classes

400 200 0 50000 30000 10000 0

100

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: Routines and classes with specification

inv

pre post0

GoboKernel: Routines and classes with specification

4000

3000

2000

1000

0

# routines or classes

# routines or classes

250

200

150

100

50

0

200

400 revision

600

800

inv

pre post0 100 revision

200

300

Quickgraph: Routines and classes with specification

pre post inv

Boogie: Routines and classes with specification 12 10 8 6 4 2 0 2000 1500 1000 pre post0

inv

pre post0 40 revision

60

80

50 revision

100

150

100

Rxx: Routines and classes with specification

20

CCI: Routines and classes with specification

pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: Routines and classes with specification

pre post inv

Dafny: Routines and classes with specification

Fig. 78. Flat: # routines and classes with specification (C# projects)

inv

500 0

# routines or classes # routines or classes

500 400 300

350 300 250 200 150 100 50

200 100 0 8 6 4

# routines or classes # routines or classes 2 0

101

# routines or classes

inv

pre 0 post

10

20 revision

30

40

50

Labs: Routines and classes with specification

# routines or classes

150 100 50 0 60 50 40 30 20

100

200 revision

300

400

5

10

15

20

25

30

Logging: Routines and classes with specification

pre 0 post

inv

pre 0 post

DirectVCGen: Routines and classes with specification

pre 0 post

inv

pre 0 post revision

400

600

800

5

10

15

20

25

30

RCC: Routines and classes with specification

200

ESCJava: Routines and classes with specification

inv revision

revision

inv

100

200 revision

300

400

50 revision

100

150

Umbra: Routines and classes with specification

pre post0

inv

pre post0

JavaFE: Routines and classes with specification

Fig. 79. Flat: # routines and classes with specification (Java projects)

inv

10 0

# routines or classes

2500 2000 1500 1000 500 0 600 500 400 300

# routines or classes # routines or classes 100 0

200

1000 800 600 400 200 25 20 15 10

# routines or classes # routines or classes

5 0

102

2.0

50

100 revision

150

200

250

300

inv

pre post0 50

100 revision

150

200

250

8 6

200

400

800

revision

600

1000

1200

inv

pre post0 20

40 revision

60

80

100

120

GoboTime: Specification clauses per routine or class

inv

pre post0

EiffelBase: Specification clauses per routine or class

50

revision

100

150

200

inv

pre post0

50

revision

100

150

200

GoboUtility: Specification clauses per routine or class

inv

pre post0

EiffelProgramAnalysis: Specification clauses per routine or class

Fig. 80. Flat: Specification clauses per routine or class (Eiffel projects)

GoboStructure: Specification clauses per routine or class

inv

pre post0

AutoTest: Specification clauses per routine or class

0.20 0.15 0.10

1.4 1.2 1.0 0.8 0.6 0.4

1.5

1.0

0.5

# clauses per routine or class

# clauses per routine or class

7

6

5

4

3

2

1

# clauses per routine or class # clauses per routine or class

4 2 2.0 1.5 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

0.05 0.00 1.5 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

0.2 0.0 7 6 5 4 3 2 1 0

103

100

200

400 revision

300

500

600

inv

pre post0

200

revision

400

600

800

GoboXML: Specification clauses per routine or class

inv

pre post0

GoboKernel: Specification clauses per routine or class

3.0

2.5

200

400 revision

600

800

inv

pre post0 100 revision

200

300

Quickgraph: Specification clauses per routine or class

pre post inv

Boogie: Specification clauses per routine or class 0.06 0.05 0.04 0.03

2.0

1.5

1.0

0.5

# clauses per routine or class

# clauses per routine or class

0.0

0.3

0.2

0.1

0.0

# clauses per routine or class

inv

pre post0

inv

pre post0 40 revision

60

80

100

50 revision

100

150

Rxx: Specification clauses per routine or class

20

CCI: Specification clauses per routine or class

1.5 pre 0 post inv

100

150

250

revision

200

300

350

400

10

20

revision

30

40

50

60

shweet: Specification clauses per routine or class

pre post inv

Dafny: Specification clauses per routine or class

Fig. 81. Flat: Specification clauses per routine or class (C# projects)

# clauses per routine or class

0.02 0.01 0.00 2.0 1.5 1.0 0.5 0.0

# clauses per routine or class # clauses per routine or class

1.4 1.2 1.0 0.8 0.6 0.4 0.2

1.0 0.5 0.0 0.6 0.5 0.4 0.3 0.2 0.1 0.0

104

# clauses per routine or class

inv

pre 0 post

10

20 revision

30

40

50

Labs: Specification clauses per routine or class

100

200 revision

300

400

inv

pre 0 post 5

10 revision

15

20

30 inv

pre 0 post

inv

pre 0 post revision

400

600

800

5

10

revision

15

20

25

30

RCC: Specification clauses per routine or class

200

ESCJava: Specification clauses per routine or class

inv

100

200 revision

300

400

50 revision

100

150

Umbra: Specification clauses per routine or class

pre post0

inv

pre post0

JavaFE: Specification clauses per routine or class

Fig. 82. Flat: Specification clauses per routine or class (Java projects)

25

Logging: Specification clauses per routine or class

inv

pre 0 post

DirectVCGen: Specification clauses per routine or class

0.15

1.0 0.8 0.6

3.0 2.5 2.0 1.5

0.10 0.05 0.00

# clauses per routine or class # clauses per routine or class

0.6 0.5 0.4 0.3 0.2 0.1

# clauses per routine or class # clauses per routine or class

0.4 0.2 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

# clauses per routine or class # clauses per routine or class

1.0 0.5 0.5 0.4 0.3 0.2 0.1 0.0

105

inv

pre post0 50

100 revision

150

200

250

300

# clauses per routine or class

inv

pre post0 200

400

800

revision

600

1000

1200 inv

pre post0 50

revision

100

150

200

inv

pre post0

100

200

400 revision

300

500

600

EiffelBase: Average specification strength per routine or class EiffelProgramAnalysis: Average specification strength per routine or GoboKernel: class Average specification strength per routine or class

# clauses per routine or class

AutoTest: Average specification strength per routine or class

# clauses per routine or class

inv

pre post0 50

100 revision

150

200

# clauses per routine or class

8

7

6

5

4

3

2

5 4 3 2 1 0 inv

pre post0 20

40 revision

60

80

100

120

3.0 2.5 2.0 1.5 1.0 0.5 0.0

# clauses per routine or class

# clauses per routine or class

inv

pre post0

50

revision

100

150

Fig. 83. Flat: Average specification strength per routine or class (Eiffel projects)

250

200

# clauses per routine or class

inv

pre post0

200

revision

400

600

800

GoboStructure: Average specification strength per routine or classGoboTime: Average specification strength per routine or class GoboUtility: Average specification strength per routine or class GoboXML: Average specification strength per routine or class

# clauses per routine or class

3.5

3.0

2.5

2.0

1.5

12 10 8 6 4 2

1.5 1.0 0.5 0.0

3.0 2.5 2.0 1.5 1.0 0.5 0.0 8 6 4 2 0

106

# clauses per routine or class

5

200

400 revision

600

800

inv

pre post0 100 revision

200

300

Quickgraph: Average specification strength per routine or class

pre post inv

Boogie: Average specification strength per routine or class

20

40 revision

60

80

100

50 revision

100

150

100

150

250

revision

200

300

350

400

inv

pre 0 post

10

20

revision

30

40

50

60

shweet: Average specification strength per routine or class

pre post inv

Dafny: Average specification strength per routine or class

Fig. 84. Flat: Average specification strength per routine or class (C# projects)

inv

pre post0

Rxx: Average specification strength per routine or class

inv

pre post0

CCI: Average specification strength per routine or class 4 3 2

2.0 1.5 1.0 0.5

4

3

2

1

2.0

1.5

1.0

0.5

0.0

# clauses per routine or class

# clauses per routine or class # clauses per routine or class

0.0 8 6 4 2 0

# clauses per routine or class # clauses per routine or class

inv

pre 0 post

10

20 revision

30

40

50

Labs: Average specification strength per routine or class

3.0 2.5 2.0 1.5 1.0

1 0 2.0 1.5 1.0 0.5 0.0

107

# clauses per routine or class

100

200 revision

300

400

inv

pre 0 post 5

10 revision

15

25

30

200 revision

400

600

800

inv

pre 0 post 5

10

revision

15

20

25

30

RCC: Average specification strength per routine or class

inv

pre 0 post

100

200 revision

300

400

inv

pre post0

50 revision

100

150

Umbra: Average specification strength per routine or class

inv

pre post0

JavaFE: Average specification strength per routine or class

Fig. 85. Flat: Average specification strength per routine or class (Java projects)

20

Logging: Average specification strength per routine or class

inv

pre 0 post

DirectVCGen: Average specification strength per routine or class ESCJava: Average specification strength per routine or class

# clauses per routine or class

2.0 1.5 1.0 0.5 0.0 2.0

6 5 4

5 4 3 2

1.8 1.6 1.4 1.2 1.0

# clauses per routine or class

# clauses per routine or class # clauses per routine or class

1 6 5 4 3 2 1

# clauses per routine or class # clauses per routine or class

3 2 6 5 4 3 2 1 0

108

# routines

# routines

100

150 revision

200

250

300

inv strong

pre weak pre strong post weak post strong 0 inv weak 50

100 revision

150

200

250

GoboStructure: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 50 inv weak

AutoTest: Weakening and strengthening 200 150

400

800

revision

600

1000

1200

40

60 revision

80

100

120

50

100 revision

150

200

inv strong

pre weak pre strong post weak post strong 0 inv weak

50

revision

100

150

200

GoboUtility: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 inv weak

EiffelProgramAnalysis: Weakening and strengthening

Fig. 86. Flat: Weakening and strengthening (Eiffel projects)

pre weak pre strong post weak post strong 20 inv 0 weak inv strong

GoboTime: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 200 inv weak

EiffelBase: Weakening and strengthening 70 60 50 40

30 25 20

20

15

10

5

0

120

100

80

60

40

20

0

# routines # routines

100 50 0 14 12 10 8 6 4 2 0

# routines # routines

30 20 10 0 8 6 4 2 0

# routines # routines

15 10 5 0 1500 1000 500 0

109

200

400 revision

300

500

600

inv strong

pre weak pre strong post weak post strong 0 inv weak

200

revision

400

600

800

GoboXML: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong 0 100 inv weak

GoboKernel: Weakening and strengthening

# routines

# routines

400 revision

600

800

inv strong

pre weak pre strong post weak post strong 0 inv weak 100 revision

200

300

Quickgraph: Weakening and strengthening

inv strong

pre weak pre strong post weak post strong inv weak 200

Boogie: Weakening and strengthening

40 revision

60

80

50 revision

100

Rxx: Weakening and strengthening

20

150

100

150

250

revision

200

300

350

20

revision

30

40

50

shweet: Weakening and strengthening

pre weak pre strong post weak post strong 10 inv0weak inv strong

inv strong

pre weak pre strong post weak post strong inv weak 100

Dafny: Weakening and strengthening

Fig. 87. Flat: Weakening and strengthening (C# projects)

pre weak pre strong post weak post strong inv 0 weak inv strong

pre weak pre strong post weak post strong inv 0weak inv strong

CCI: Weakening and strengthening

20 15

3.0 2.5 2.0 1.5

250

200

150

100

50

0

80

60

40

20

0

# routines # routines

1.0 0.5 0.0 15 10 5 0

# routines # routines

60

400

150 100 50 0

10 5 0 2.0 1.5 1.0 0.5 0.0

110

# routines

pre weak pre strong post weak post strong inv0weak inv strong

10

20 revision

30

40

Labs: Weakening and strengthening

50

# routines # routines

100

200 revision

300

pre weak pre strong post weak post strong 5 inv0 weak inv strong 10 revision

15

20

25

30

400

200 revision

400

600

800

10

revision

15

20

25

RCC: Weakening and strengthening

pre weak pre strong post weak post strong inv0 weak 5 inv strong

pre weak pre strong post weak post strong inv 0weak inv strong

ESCJava: Weakening and strengthening

30

200 revision

300

50 revision

100

Umbra: Weakening and strengthening

pre weak pre strong post weak post strong inv 0 weak inv strong

inv strong

100

JavaFE: Weakening and strengthening

pre weak pre strong post weak post strong 0 inv weak

Fig. 88. Flat: Weakening and strengthening (Java projects)

Logging: Weakening and strengthening

pre weak pre strong post weak post strong inv0 weak inv strong

DirectVCGen: Weakening and strengthening 300 250 200

70 60 50

10 8 6 4 2 0 20 15 10 5 0

# routines # routines

150 100 50 0 600 500 400 300 200 100 0

# routines # routines

40 30 20 10 0 20 15 10 5 0

111

150

400

% routines

% routines

0.80

0.75

0.70

0.65

0.60

0.55

0.9

0.8

0.7

0.6

0.5

0.4

100

150 revision

200

250

300

0 public non−public

50

100 revision

150

200

400

800

revision

600

1000

1200

0 20 public non−public

40

60

80

100

GoboTime: % routines with pre or post

0 200 public non−public

EiffelBase: % routines with pre or post

revision

120

50

100 revision

150

0 public non−public

50

revision

100

150

200

200

GoboUtility: % routines with pre or post

0 public non−public

EiffelProgramAnalysis: % routines with pre or post

Fig. 89. Flat: % routines with specification: public vs nonpublic (Eiffel projects)

250

GoboStructure: % routines with pre or post

0 50 public non−public

AutoTest: % routines with pre or post 0.7 0.6 0.5 0.4 0.3 1.0 0.8 0.6 0.2

% routines % routines

0.5 0.4

0.0

0.4

% routines % routines

0.3 0.2 0.1 0.0 1.0 0.8 0.6 0.4 0.2 0.0

% routines % routines

1.0 0.8 0.6 0.4 0.2 0.0 0.6 0.4 0.2 0.0

112

200

400 revision

300

500

600

0 public non−public

200

revision

400

600

800

GoboXML: % routines with pre or post

0 100 public non−public

GoboKernel: % routines with pre or post

% routines

% routines

0.80

0.75

0.70

0.65

0.60

0.55

0.50

0.20

0.15

400 revision

600

800

0 public non−public

100

200

300

Quickgraph: % routines with pre or post

200 public non−public

Boogie: % routines with pre or post

revision

0 public non−public

0 public non−public

40 revision

60

80

50 revision

100

Rxx: % routines with pre or post

20

CCI: % routines with pre or post

150

100

150

250

revision

200

300

350

0 10 public non−public

20

revision

30

40

50

shweet: % routines with pre or post

public 100 non−public

Dafny: % routines with pre or post

Fig. 90. Flat: % routines with specification: public vs nonpublic (C# projects)

0.05

0.7 0.6 0.5 0.4

0.15

0.00

0.10

% routines % routines

0.10 0.05 0.00 1.0 0.8 0.6 0.4 0.2 0.0

% routines % routines

60

400

0.60 0.55 0.50 0.45 0.40 0.35 0.30

0.3 0.2 0.4 0.3 0.2 0.1 0.0

113

% routines

0 public non−public

10

20 revision

30

40

Labs: % routines with pre or post

50

% routines % routines

100

200 revision

300

10 revision

15

25

30

400

0 5 public non−public

0 public non−public revision

400

600

10

revision

15

20

25

RCC: % routines with pre or post

200

800

ESCJava: % routines with pre or post

30

0.7 0 public non−public

0 public non−public

200 revision

300

50 revision

100

Umbra: % routines with pre or post

100

JavaFE: % routines with pre or post

Fig. 91. Flat: % routines with specification: public vs nonpublic (Java projects)

20

Logging: % routines with pre or post

0 5 public non−public

0 public non−public

DirectVCGen: % routines with pre or post 0.7 0.6 0.5

0.20 0.15 0.10 0.05 0.00 0.5 0.4 0.3 0.2 0.1

% routines % routines

0.4 0.3 0.2 0.7 0.6 0.5 0.4 0.3 0.2 0.1

% routines % routines

0.6 0.5 0.4 0.25 0.20 0.15 0.10 0.05 0.00

114

150

400

25

15

10

5

0

# classes

−A weak

+A same +A strong +A weak −A same 0 −A strong 50

100

150 revision

200

250

300 −A weak

+A same +A strong +A weak −A same 0 200 −A strong 400

800

revision

600

1000

1200

−A weak

+A same +A strong +A weak −A same 0 −A strong 50

100

150 revision

200

250

20

40

60 revision

80

100

120

100 revision

−A weak

+A same +A strong +A weak −A same 0 −A strong

50

revision

100

Fig. 92. Flat: Weakening and strengthening of class invariants (Eiffel projects)

+A same +A strong +A weak −A same −A 0 strong −A weak

50

150

200

150

200

GoboUtility: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 −A strong

200

400 revision

300

500

600

−A weak

+A same +A strong +A weak −A same 0 −A strong

200

revision

400

600

800

GoboXML: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 100 −A strong

EiffelBase: Weakening and strengthening of class invariants EiffelProgramAnalysis: Weakening and strengthening of class invariants GoboKernel: Weakening and strengthening of class invariants 80

AutoTest: Weakening and strengthening of class invariants

# classes # classes

GoboStructure: Weakening and strengthening of class invariants GoboTime: Weakening and strengthening of class invariants

# classes

20

15

10

5

0

25 20 15 10 5 0 3.0 2.5 2.0 1.5 1.0 0.5 0.0

2.0 1.5 1.0 0.5

60 40 20 0 1.0 0.8 0.6 0.4 0.2 0.0

# classes # classes

# classes # classes

0.0 250 200 150 100 50 0

115

# classes

400 revision

600

800

−A weak

+A same +A strong +A weak −A same 0 −A strong 100 revision

200

300

Quickgraph: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same −A strong200

Boogie: Weakening and strengthening of class invariants

1.0

0.5

20

40 revision

60

80

100

1.0

50 revision

100

150

150

250

revision

200

300

350

400

+A same +A strong +A weak −A same −A0strong −A weak

10

20

revision

30

40

50

60

shweet: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same −A strong100

Dafny: Weakening and strengthening of class invariants

Fig. 93. Flat: Weakening and strengthening of class invariants (C# projects)

+A same +A strong +A weak −A same −A 0 strong −A weak

Rxx: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A 0strong −A weak

CCI: Weakening and strengthening of class invariants 1.0 0.5 0.0

0.5 0.0 −0.5

0.0

−0.5

−1.0

1.0

0.5

0.0

−0.5

−1.0

# classes

# classes # classes

−1.0 1.0 0.5 0.0 −0.5 −1.0

# classes # classes

1.0 0.5 0.0 −0.5 −1.0

−0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

116

# classes

+A same +A strong +A weak −A same −A0strong −A weak

10

20 revision

30

40

50

Labs: Weakening and strengthening of class invariants

# classes # classes

100

200 revision

300

400

+A same +A strong +A weak −A same −A0 strong −A weak 5

10 revision

15

25

30

200 revision

400

600

800

+A same +A strong +A weak −A same −A0 strong −A weak 5

10

revision

15

20

25

30

RCC: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A 0strong −A weak

ESCJava: Weakening and strengthening of class invariants

100

200 revision

300

400

−A weak

+A same +A strong +A weak −A same 0 −A strong

50 revision

100

150

Umbra: Weakening and strengthening of class invariants

−A weak

+A same +A strong +A weak −A same 0 −A strong

JavaFE: Weakening and strengthening of class invariants

Fig. 94. Flat: Weakening and strengthening of class invariants (Java projects)

20

Logging: Weakening and strengthening of class invariants

+A same +A strong +A weak −A same −A0 strong −A weak

DirectVCGen: Weakening and strengthening of class invariants 1.0 0.5

1.0 0.5

1.0 0.5 0.0 −0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

# classes # classes

0.0 −0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

# classes # classes

0.0 −0.5 −1.0 1.0 0.5 0.0 −0.5 −1.0

117

13

Data About the Control Group

The following tables report the same measures as those used in the main analysis, but target 10 student projects discussed at the end of Section 6.

118

# CLASSES % CLASSES INV # ROUTINES % ROUTINES SPEC % ROUTINES PRE % ROUTINES POST AVG ROUTINES PRE AVG ROUTINES POST Project m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m dose 76 0 16 28 6 0 0.53 1 0.12 0 34 86 26 0 0.36 1 0.18 0 0.29 1 0.15 0 0.23 1 0.18 0 1.46 2.12 0.23 0 2 3 0.57 0 dose 77 0 24 25 6 0 0.83 0.88 0.17 0 150 193 57 0 0.73 0.88 0.14 0 0.26 0.36 0.1 0 0.61 0.88 0.15 0 1.52 1.71 0.45 0 4.07 5.2 1.36 0 dose 78 0 33 35 8 0 0.37 0.41 0.08 0 93 96 25 0 0.12 0.16 0.02 0 0.06 0.12 0.02 0 0.1 0.12 0.04 0 1.67 1.8 0.32 0 1.22 1.56 0.51 0 dose 80 0 9 9 1 0 0.25 0.44 0.06 0 35 43 12 0 0.39 0.83 0.13 0 0.21 0.83 0.18 0 0.28 0.58 0.08 0 1.5 1.5 0.25 0 1.5 1.69 0.23 0 dose 86 0 37 79 22 0 0.28 1 0.19 0 172 406 107 0 0.49 0.68 0.15 0 0.41 0.56 0.12 0 0.24 0.3 0.08 0 1.4 1.64 0.19 0 1.55 2.44 0.44 0 dose 87 0 14 18 4 0 0.2 0.67 0.11 0 134 163 46 0 0.49 0.84 0.17 0 0.37 0.59 0.1 0 0.22 0.59 0.13 0 1.44 1.72 0.34 0 1.81 2.17 0.53 0 dose 88 0 26 32 4 0 0.74 0.85 0.14 0 229 371 87 0 0.66 0.75 0.15 0 0.59 0.71 0.19 0 0.51 0.68 0.14 0 1.66 1.88 0.48 0 1.6 1.84 0.31 0 dose 89 0 16 16 2 0 0.44 0.9 0.14 0 89 132 39 0 0.36 0.48 0.1 0 0.34 0.44 0.09 0 0.22 0.44 0.09 0 1.16 1.83 0.32 0 1.42 1.6 0.34 0 dose 90 3 12 16 3 0 0.27 0.62 0.1 0 51 131 40 0 0.38 1 0.14 0 0.18 1 0.12 0 0.32 1 0.13 0 1.22 1.44 0.29 0 1.06 1.2 0.27 0 dose 92 19 59 61 4 0 0.05 0.07 0.01 24 456 562 122 0.27 0.31 0.92 0.11 0.17 0.22 0.39 0.02 0.11 0.12 0.78 0.1 1 1.17 1.37 0.04 1 1.02 1.03 0 0

Table 95. Specification overall statistics with non-flattened classes.

M 3.12 2.46 1.92 5 3 2 2.2 1.14 1.8 3

AVG INV

µ 2.88 1.81 1.92 1.67 1.57 1 1.94 1.14 1 3

σ 0.61 0.41 0.4 1.37 0.44 0.47 0.37 0.08 0.15 0.3

119

120

Table 96. Specification overall statistics with flattened classes.

# CLASSES % CLASSES INV # ROUTINES % ROUTINES SPEC % ROUTINES PRE % ROUTINES POST AVG ROUTINES PRE AVG ROUTINES POST AVG INV Project m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ dose 76 0 16 28 6 0 0.79 1 0.14 0 46 122 37 0 0.39 1 0.13 0 0.33 1 0.11 0 0.2 1 0.16 0 1.29 1.91 0.21 0 1.71 2.5 0.46 0 2.33 2.67 0.45 dose 77 0 24 25 6 0 0.88 0.93 0.16 0 184 228 65 0 0.69 0.91 0.15 0 0.25 0.34 0.1 0 0.57 0.91 0.17 0 1.42 1.62 0.42 0 3.71 4.78 1.27 0 2 2.58 0.43 dose 78 0 33 35 8 0 0.69 0.72 0.2 0 156 162 43 0 0.31 0.34 0.09 0 0.25 0.27 0.07 0 0.26 0.28 0.11 0 2 2.1 0.47 0 1.41 1.5 0.55 0 2.52 2.54 0.63 dose 80 0 9 9 1 0 0.25 0.44 0.06 0 35 44 12 0 0.38 0.83 0.13 0 0.21 0.83 0.18 0 0.28 0.58 0.08 0 1.5 1.5 0.25 0 1.5 1.69 0.23 0 1.67 5 1.37 dose 86 0 37 79 22 0 0.56 1 0.19 0 206 574 169 0 0.49 0.6 0.11 0 0.31 0.51 0.1 0 0.29 0.35 0.1 0 1.44 1.7 0.2 0 1.32 2.44 0.47 0 1.46 3 0.44 dose 87 0 14 18 4 0 0.2 0.67 0.11 0 134 163 46 0 0.49 0.84 0.17 0 0.37 0.59 0.1 0 0.22 0.59 0.13 0 1.44 1.72 0.34 0 1.81 2.17 0.53 0 1.33 2 0.47 dose 88 0 26 32 4 0 0.78 0.9 0.15 0 310 449 106 0 0.67 0.73 0.15 0 0.57 0.64 0.17 0 0.55 0.73 0.14 0 1.64 1.87 0.48 0 1.53 1.77 0.29 0 3.43 3.63 0.68 dose 89 0 16 16 2 0 0.5 0.92 0.15 0 89 132 39 0 0.53 0.84 0.14 0 0.5 0.81 0.14 0 0.26 0.56 0.11 0 1.39 1.79 0.25 0 1.33 1.42 0.21 0 1.12 1.12 0.08 dose 90 3 12 16 3 0 0.27 0.62 0.1 0 89 196 59 0 0.46 1 0.18 0 0.29 1 0.13 0 0.36 1 0.14 0 1.08 1.33 0.24 0 1.24 1.41 0.31 0 1 3 0.33 dose 92 19 59 61 4 0 0.1 0.14 0.02 56 862 1029 206 0.42 0.48 0.96 0.12 0.08 0.27 0.39 0.05 0.2 0.22 0.86 0.11 1 1.36 1.41 0.07 1 1 1.06 0.01 0 3 3 0.3

Project dose 76 dose 77 dose 78 dose 80 dose 86 dose 87 dose 88 dose 89 dose 90 dose 92

lang Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel

LOC # CLASSES % CLASSES INV # ROUTINES % ROUTINES SPEC % ROUTINES PRE % ROUTINES POST rev age m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ 151 8 0 2051 3494 1055 0 16 28 6 0 0.53 1 0.12 0 34 86 26 0 0.36 1 0.18 0 0.29 1 0.15 0 0.23 1 0.18 70 8 0 3486 4349 1177 0 24 25 6 0 0.83 0.88 0.17 0 150 193 57 0 0.73 0.88 0.14 0 0.26 0.36 0.1 0 0.61 0.88 0.15 73 8 0 2908 3208 930 0 33 35 8 0 0.37 0.41 0.08 0 93 96 25 0 0.12 0.16 0.02 0 0.06 0.12 0.02 0 0.1 0.12 0.04 143 7 0 1556 1944 516 0 9 9 1 0 0.25 0.44 0.06 0 35 43 12 0 0.39 0.83 0.13 0 0.21 0.83 0.18 0 0.28 0.58 0.08 170 8 0 3086 8827 2519 0 37 79 22 0 0.28 1 0.19 0 172 406 107 0 0.49 0.68 0.15 0 0.41 0.56 0.12 0 0.24 0.3 0.08 235 7 0 2358 4460 1287 0 14 18 4 0 0.2 0.67 0.11 0 134 163 46 0 0.49 0.84 0.17 0 0.37 0.59 0.1 0 0.22 0.59 0.13 113 8 0 4320 8786 2350 0 26 32 4 0 0.74 0.85 0.14 0 229 371 87 0 0.66 0.75 0.15 0 0.59 0.71 0.19 0 0.51 0.68 0.14 216 6 0 4370 7290 2093 0 16 16 2 0 0.44 0.9 0.14 0 89 132 39 0 0.36 0.48 0.1 0 0.34 0.44 0.09 0 0.22 0.44 0.09 155 7 30 833 2764 752 3 12 16 3 0 0.27 0.62 0.1 0 51 131 40 0 0.38 1 0.14 0 0.18 1 0.12 0 0.32 1 0.13 292 6 522 11374 16372 4243 19 59 61 4 0 0.05 0.07 0.01 24 456 562 122 0.27 0.31 0.92 0.11 0.17 0.22 0.39 0.02 0.11 0.12 0.78 0.1

Table 97. Project specification statistics for non-flattened classes.

121

122

Project dose 76 dose 77 dose 78 dose 80 dose 86 dose 87 dose 88 dose 89 dose 90 dose 92

lang Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel Eiffel

LOC rev age m µ M 151 8 0 2051 3494 70 8 0 3486 4349 73 8 0 2908 3208 143 7 0 1556 1944 170 8 0 3086 8827 235 7 0 2358 4460 113 8 0 4320 8786 216 6 0 4370 7290 155 7 30 833 2764 292 6 522 11374 16372

Table 98. Project specification statistics for flattened classes.

# CLASSES % CLASSES INV # ROUTINES % ROUTINES SPEC % ROUTINES PRE % ROUTINES POST σ m µM σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ 1055 0 16 28 6 0 0.79 1 0.14 0 46 122 37 0 0.39 1 0.13 0 0.33 1 0.11 0 0.2 1 0.16 1177 0 24 25 6 0 0.88 0.93 0.16 0 184 228 65 0 0.69 0.91 0.15 0 0.25 0.34 0.1 0 0.57 0.91 0.17 930 0 33 35 8 0 0.69 0.72 0.2 0 156 162 43 0 0.31 0.34 0.09 0 0.25 0.27 0.07 0 0.26 0.28 0.11 516 0 9 9 1 0 0.25 0.44 0.06 0 35 44 12 0 0.38 0.83 0.13 0 0.21 0.83 0.18 0 0.28 0.58 0.08 2519 0 37 79 22 0 0.56 1 0.19 0 206 574 169 0 0.49 0.6 0.11 0 0.31 0.51 0.1 0 0.29 0.35 0.1 1287 0 14 18 4 0 0.2 0.67 0.11 0 134 163 46 0 0.49 0.84 0.17 0 0.37 0.59 0.1 0 0.22 0.59 0.13 2350 0 26 32 4 0 0.78 0.9 0.15 0 310 449 106 0 0.67 0.73 0.15 0 0.57 0.64 0.17 0 0.55 0.73 0.14 2093 0 16 16 2 0 0.5 0.92 0.15 0 89 132 39 0 0.53 0.84 0.14 0 0.5 0.81 0.14 0 0.26 0.56 0.11 752 3 12 16 3 0 0.27 0.62 0.1 0 89 196 59 0 0.46 1 0.18 0 0.29 1 0.13 0 0.36 1 0.14 4243 19 59 61 4 0 0.1 0.14 0.02 56 862 1029 206 0.42 0.48 0.96 0.12 0.08 0.27 0.39 0.05 0.2 0.22 0.86 0.11

−A inv weak vs same

−A inv strong vs same

2.50E+00 1.18E-02 -3.00E+00 -9.94E-01 2.50E+00 1.18E-02 -3.00E+00 -9.94E-01

+A inv strong vs same

post weak vs strong

3.50E+00 1.40E-02 -1.00E+00 -4.39E-01 3.50E+00 1.40E-02 -1.00E+00 -4.39E-01

+A inv weak vs same

NE pre weak vs strong

1.00E+00 1.26E-02 -3.50E+00 -7.05E-01 1.00E+00 1.26E-02 -3.50E+00 -7.05E-01

NE inv weak vs strong

pre weak vs strong

3.05E+01 3.74E-01 2.50E+00 3.93E-01 3.05E+01 3.74E-01 2.50E+00 3.93E-01

inv weak vs strong

NE chang vs unch spec

0.00E+00 1.95E-03 -4.10E+01 -1.19E+00 0.00E+00 1.95E-03 -4.10E+01 -1.19E+00

NE post weak vs strong

chang vs unch spec V p ∆(µ) d V p ∆(µ) d

0.00E+00 1.50E+00 4.18E-02 0.00E+00 -2.00E+00 0.00E+00 -7.04E-01 0.00E+00 1.50E+00 4.18E-02 0.00E+00 -2.00E+00 0.00E+00 -7.04E-01

2.00E+00 8.90E-02 -5.00E-01 -4.66E-01 2.00E+00 8.90E-02 -5.00E-01 -4.66E-01

0.00E+00 5.86E-03 -1.50E+01 -1.41E+00 0.00E+00 5.86E-03 -1.50E+01 -1.41E+00

0.00E+00 5.86E-03 -1.40E+01 -1.33E+00 0.00E+00 5.86E-03 -1.40E+01 -1.33E+00

0.00E+00 5.83E-03 -4.50E+00 -1.14E+00 0.00E+00 5.83E-03 -4.50E+00 -1.14E+00

0.00E+00 5.76E-03 -4.50E+00 -1.19E+00 0.00E+00 5.76E-03 -4.50E+00 -1.19E+00

−A inv weak vs same

−A inv strong vs same

0.00E+00 5.89E-03 -8.00E+00 -9.48E-01 0.00E+00 5.89E-03 -8.00E+00 -9.48E-01

+A inv strong vs same

0.00E+00 5.67E-03 -2.00E+00 -5.63E-01 0.00E+00 5.67E-03 -2.00E+00 -5.63E-01

+A inv weak vs same

post weak vs strong

6.00E+00 3.22E-02 -8.00E+00 -7.41E-01 6.00E+00 3.22E-02 -8.00E+00 -7.41E-01

NE inv weak vs strong

NE pre weak vs strong

2.80E+01 5.54E-01 2.50E+00 2.61E-01 2.80E+01 5.54E-01 2.50E+00 2.61E-01

inv weak vs strong

pre weak vs strong

0.00E+00 1.95E-03 -1.31E+02 -9.99E-01 0.00E+00 1.95E-03 -1.31E+02 -9.99E-01

NE post weak vs strong

NE chang vs unch spec

V p ∆(µ) d V p ∆(µ) d

chang vs unch spec

Table 99. Change analysis with majority for non-flattened (top) and flattened (bottom) classes.

0.00E+00 5.00E+00 4.36E-02 0.00E+00 -3.00E+00 0.00E+00 -8.55E-01 0.00E+00 5.00E+00 4.36E-02 0.00E+00 -3.00E+00 0.00E+00 -8.55E-01

2.50E+00 3.32E-02 -1.00E+00 -5.94E-01 2.50E+00 3.32E-02 -1.00E+00 -5.94E-01

0.00E+00 1.95E-03 -2.00E+01 -1.34E+00 0.00E+00 1.95E-03 -2.00E+01 -1.34E+00

0.00E+00 5.89E-03 -1.70E+01 -1.28E+00 0.00E+00 5.89E-03 -1.70E+01 -1.28E+00

0.00E+00 5.86E-03 -7.00E+00 -1.26E+00 0.00E+00 5.86E-03 -7.00E+00 -1.26E+00

0.00E+00 5.86E-03 -8.00E+00 -1.29E+00 0.00E+00 5.86E-03 -8.00E+00 -1.29E+00

Table 100. Change analysis for non-flattened (top) and flattened (bottom) classes.

123

124

Project dose 76 dose 77 dose 78 dose 80 dose 86 dose 87 dose 88 dose 89 dose 90 dose 92 Overall

τ 0.89 0.86 0.48 1.00 0.35 1.00 0.89 1.00 1.00 1.00 0.87

p 0.00E+00 0.00E+00 4.44E-16 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 9.70E-04 0.00E+00

τ 0.76 0.81 0.78 1.00 0.68 1.00 0.89 0.76 0.76 0.14 0.62

p 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 3.15E-10 0.00E+00 0.00E+00

τ 0.95 0.94 0.80 1.00 0.87 1.00 0.83 0.77 0.38 0.41 0.87

p 0.00E+00 0.00E+00 0.00E+00 0.00E+00 3.42E-05 0.00E+00 6.66E-16 0.00E+00 0.00E+00 0.00E+00 0.00E+00

τ 0.66 0.79 0.86 1.00 0.23 0.85 0.60 1.00 1.00 1.00 0.79

CLASSES W / INV AVG PRE STRENGTH AVG POST STRENGTH AVG INV STRENGTH

p 0.00E+00 0.00E+00 7.58E-08 0.00E+00 7.07E-11 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00

%

Table 101. Correlation of measures in flat and non-flat classes.

τ 0.67 0.96 0.53 0.94 0.80 1.00 0.93 0.58 0.71 0.71 0.59

ROUTINES W / SPEC

p 0.00E+00 0.00E+00 1.05E-09 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00 0.00E+00

%

Project dose 76 dose 77 dose 78 dose 80 dose 86 dose 87 dose 88 dose 89 dose 90 dose 92 Overall

% SPEC /# ROUTINES p τ 8.31E-31 -0.66 7.21E-08 -0.45 6.71E-01 -0.04 4.91E-20 -0.57 1.61E-08 -0.30 2.06E-06 -0.21 4.02E-01 0.05 1.40E-14 -0.36 2.08E-09 -0.33 9.80E-67 -0.69 5.76E-06 -0.08

# SPEC /# ROUTINES p τ 0.00E+00 0.65 0.00E+00 0.94 2.82E-04 0.34 0.00E+00 0.56 0.00E+00 0.79 0.00E+00 0.51 0.00E+00 0.75 0.00E+00 0.95 0.00E+00 0.84 0.00E+00 0.92 0.00E+00 0.79

% INV /# CLASSES p τ 8.79E-45 -0.85 1.77E-07 0.49 5.70E-04 0.31 3.38E-04 -0.28 1.67E-15 -0.43 6.42E-24 -0.50 1.19E-02 -0.19 8.79E-40 -0.76 7.74E-26 -0.66 3.89E-44 -0.67 1.76E-27 -0.19

# INV /# CLASSES p τ 0.00E+00 0.89 0.00E+00 0.91 0.00E+00 0.82 9.36E-01 0.01 0.00E+00 0.90 3.93E-05 0.22 0.00E+00 0.74 1.68E-30 -0.68 4.00E-15 0.54 3.91E-03 -0.15 0.00E+00 0.30

AVG /% PRE

p 1.11E-15 5.04E-03 2.90E-10 2.40E-25 0.00E+00 1.93E-02 8.71E-01 0.00E+00 3.69E-01 5.10E-09 0.00E+00

τ 0.47 0.24 -0.61 -0.72 0.65 -0.11 -0.01 0.47 -0.05 -0.24 0.20

AVG /% POST

p 3.20E-09 1.27E-03 1.39E-04 1.30E-04 2.49E-06 1.28E-08 1.32E-01 0.00E+00 0.00E+00 0.00E+00 0.00E+00

τ 0.34 -0.27 0.35 0.24 0.25 -0.26 -0.10 0.59 0.60 0.44 0.41

AVG /% INV

p 9.84E-01 9.03E-09 6.82E-14 9.15E-01 1.72E-13 0.00E+00 1.08E-02 6.02E-59 1.62E-05 2.00E-03 8.95E-03

τ -0.00 0.52 0.71 -0.01 -0.41 0.66 0.19 -0.93 0.31 -0.16 -0.05

Table 102. Correlation of measures of specification in non-flat classes.

Project dose 76 dose 77 dose 78 dose 80 dose 86 dose 87 dose 88 dose 89 dose 90 dose 92 Overall

% SPEC /# ROUTINES p τ 6.87E-15 -0.45 9.81E-08 -0.44 9.22E-07 0.42 7.68E-25 -0.63 6.16E-02 -0.10 2.06E-06 -0.21 8.09E-01 0.02 1.27E-47 -0.68 1.92E-13 -0.41 2.12E-118 -0.92 9.48E-01 0.00

# SPEC /# ROUTINES p τ 0.00E+00 0.84 0.00E+00 0.93 0.00E+00 0.84 0.00E+00 0.54 0.00E+00 0.96 0.00E+00 0.51 0.00E+00 0.76 0.00E+00 0.96 0.00E+00 0.73 0.00E+00 0.73 0.00E+00 0.84

% INV /# CLASSES p τ 9.71E-42 -0.82 3.10E-04 0.34 5.35E-03 0.25 3.38E-04 -0.28 2.04E-05 0.23 6.42E-24 -0.50 5.39E-05 -0.30 2.34E-39 -0.76 7.74E-26 -0.66 3.89E-44 -0.67 2.08E-10 -0.11

# INV /# CLASSES p τ 0.00E+00 0.94 0.00E+00 0.91 0.00E+00 0.94 9.36E-01 0.01 0.00E+00 0.94 3.93E-05 0.22 0.00E+00 0.73 1.39E-28 -0.66 4.00E-15 0.54 3.91E-03 -0.15 0.00E+00 0.44

AVG /% PRE

p 0.00E+00 8.46E-07 1.68E-03 8.74E-25 0.00E+00 1.93E-02 4.96E-01 0.00E+00 2.05E-03 2.16E-12 0.00E+00

Table 103. Correlation of measures of specification in flat classes.

125

τ 0.71 0.41 0.27 -0.70 0.51 -0.11 -0.04 0.62 0.18 0.29 0.23

AVG /% POST

p 5.77E-09 9.37E-04 1.79E-06 5.63E-05 3.84E-04 1.28E-08 6.50E-02 0.00E+00 3.55E-02 0.00E+00 0.00E+00

τ 0.34 -0.28 0.41 0.25 -0.19 -0.26 -0.12 0.41 0.12 0.57 0.23

AVG /% INV

p 3.70E-09 8.18E-09 2.17E-04 9.15E-01 4.42E-02 0.00E+00 1.66E-04 1.16E-58 1.62E-05 2.00E-03 4.81E-01

τ -0.35 0.54 0.34 -0.01 0.11 0.70 0.28 -0.93 0.31 -0.16 -0.01

% PRE PUBLIC / PRIVATE % POST PUBLIC / PRIVATE % SPEC PUBLIC / PRIVATE Project p τ p τ p τ dose 76 0.00E+00 0.51 0.00E+00 0.83 0.00E+00 0.48 dose 77 8.76E-10 0.53 1.93E-14 0.67 5.99E-08 0.47 dose 78 3.01E-12 -0.62 3.85E-03 0.25 2.07E-01 0.11 dose 80 3.86E-01 0.05 2.25E-06 -0.29 3.13E-01 -0.06 dose 86 0.00E+00 0.72 0.00E+00 0.62 0.00E+00 0.69 dose 87 6.68E-02 -0.10 6.69E-01 -0.02 3.17E-01 0.05 dose 88 5.89E-02 0.13 2.09E-10 0.42 8.45E-02 0.12 dose 89 2.39E-04 0.17 0.00E+00 0.45 1.55E-06 0.23 dose 90 1.79E-01 0.09 1.11E-01 -0.10 5.93E-01 -0.03 dose 92 4.66E-15 0.33 3.62E-03 -0.12 2.75E-01 0.05 Overall 0.00E+00 0.23 0.00E+00 0.21 0.00E+00 0.16 Table 104. Correlation of measures between public and private routines in non-flat classes.

% PRE PUBLIC / PRIVATE % POST PUBLIC / PRIVATE % SPEC PUBLIC / PRIVATE Project p τ p τ p τ dose 76 7.26E-01 -0.02 0.00E+00 0.79 2.30E-09 0.34 dose 77 1.11E-13 0.64 4.22E-15 0.68 1.76E-10 0.55 dose 78 1.84E-01 0.11 4.17E-03 0.25 2.43E-01 0.10 dose 80 2.09E-01 0.08 6.54E-05 -0.24 8.02E-01 0.02 dose 86 0.00E+00 0.64 0.00E+00 0.49 0.00E+00 0.66 dose 87 6.68E-02 -0.10 4.40E-01 0.04 1.49E-02 0.12 dose 88 7.62E-01 -0.02 7.97E-11 0.43 3.13E-01 0.07 dose 89 2.50E-01 -0.05 0.00E+00 0.44 1.41E-01 0.07 dose 90 8.95E-01 0.01 3.20E-05 -0.27 1.71E-06 -0.31 dose 92 0.00E+00 0.44 0.00E+00 0.66 0.00E+00 0.79 Overall 0.00E+00 0.18 0.00E+00 0.20 1.05E-11 0.12 Table 105. Correlation of measures between public and private routines in flat classes.

126

% PRE VOID Project m µ M σ m dose 76 0 0.57 1 0.09 0 dose 77 0 0.25 0.53 0.12 0 dose 78 0 0.33 0.43 0.15 0 dose 80 0 0.12 1 0.2 0 dose 86 0 0.96 0.98 0.19 0 dose 87 0 0.35 0.91 0.18 0 dose 88 0 0.94 1 0.27 0 dose 89 0 0.57 0.61 0.14 0 dose 90 0 0.78 0.9 0.22 0 dose 92 0.44 0.59 0.65 0.05 0

% PRE ∀, ∃ % POST VOID % POST ∀, ∃ % POST OLD % INV VOID % INV ∀, ∃ µ M σ m µ M σ m µ M σ m µ M σ m µ M σ m µ M σ 0 0 0 0 0.09 0.18 0.06 0 0 0 0 0 0.09 0.29 0.06 0 0.29 0.46 0.1 0 0 0 0 0 0 0 0 0.05 0.08 0.01 0 0 0 0 0 0.78 0.87 0.14 0 0.62 0.82 0.17 0 0 0 0 0 0 0 0 0.89 1 0.39 0 0 0 0 0 0 0 0 0 0.38 0.38 0.16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0.64 0.2 0 0.5 0.5 0.09 0 0 0 0 0.01 0.02 0.01 0 0.32 0.72 0.21 0 0.25 0.6 0.18 0 0 0 0 0 1 1 0.27 0 0.12 0.14 0.04 0 0 0 0 0.37 1 0.2 0 0 0 0 0 0 0.02 0 0 0 0.88 0.17 0 0 0 0 0.02 0.04 0.01 0 0.33 1 0.19 0 0 0 0 0 0 0 0 0 0.8 1 0.14 0 0 0 0 0 0 0 0 0.21 0.6 0.1 0 0 0 0 0 0.21 0.29 0.12 0 0 0 0 0 0 0 0 0 0 0 0 0.12 0.38 0.06 0 0 0 0 0 0.12 0.4 0.09 0 0 0.25 0.11 0 0 0 0 0.01 0.04 0 0 0.4 0.46 0.15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Table 106. Syntactic elements found in contracts (non-flattened classes).

127