A Runtime Monitoring Framework for Event Streams with Non-Primitive Arguments Jérôme Calvar, Raphaël Tremblay-Lessard and Sylvain Hallé Université du Québec à Chicoutimi, Canada Email:
[email protected]
Abstract—A runtime monitor is a tool that takes as input a model of some system, and observes in real time that the sequence of events produced by a run of that system follows the specification. While existing monitoring solutions generally use finite-state machines and temporal logic as their model language, the specification is ultimately tangled with handwritten, implementation-specific details which severely limit their range of application. We present a runtime monitoring platform that clearly separates the extraction of events in the running program from the specification and monitoring process. This separation allows one to cleanly monitor firstorder properties involving arbitrarily complex native program objects, while still incurring reasonable overhead. Keywords-Runtime monitoring, object-oriented programming
I. I NTRODUCTION Runtime monitoring is the process of observing an actual run of a system and dynamically checking and enforcing constraints on its execution [13]. This mechanism is more powerful than the mere verification of pre- or postconditions inside function bodies, as dependencies between sequences of method or function calls can be expressed and enforced. In the network protocols community, the technique has also been called passive testing [2], [12], [16]. Recent software monitoring efforts in have focused almost exclusively on the Java language, mostly due to the existence of aspect-oriented programming libraries such as AspectJ [10], that can easily catch appropriate method calls inside the program to monitor. Popular frameworks support a range of input languages, and have been shown to efficiently enforce method sequencing constraints with reasonable load on a program [4], [6]. The majority of available monitors treat method calls as “atomic” events and abstract away any arguments they may carry. They can model a constraint of the form “when method a() is called, eventually method b() will”, but are mostly helpless for a property like “when method a(x) is called, eventually method b(y) will, for a value y that depends on x”. Some solutions do provide support for method arguments, and allow events to carry objects from the program over to the monitor. However, while the technique works reasonably well with primitive data types (e.g. int), it becomes problematic when handling conditions on arguments that are arbitrarily complex objects. If the monitor is passed native program
objects and verifies the properties by directly querying the internal state of these objects, the specification to monitor becomes tangled with implementation-specific instructions, and hardly qualifies any longer as a formal “model” of the system. As a consequence, current monitoring frameworks suffer from a poor decoupling between the specification and verification parts of the monitoring process. To illustrate our point, in Section II we present on a simple example the context of runtime monitoring and survey the state-ofthe art. Section III then describes our runtime monitoring platform, which clearly separates the system to monitor, the monitored events generated by the system, the specification to enforce on those events, and the monitor that checks the specification. It does so by sending information about each method call through an “event formatter”, which relays an XML-like “message” to a pluggable runtime monitor class. This monitor accepts specifications expressed in a first-order extension of Linear Temporal Logic, called LTL-FO+ , that allows complex dependencies between multiple method calls. In Section IV, we show by means of an extended scenario how this framework readily handles non-primitive arguments in method calls. However, the inclusion of an event formatter into the loop is not a mere bridge between program objects and their trivial serialization into XML. One can take advantage of the formatter to record into event messages arbitrary properties about the arguments; combined with LTL-FO+ expressiveness, the platform hence allows the monitoring of correlations on the state of objects across multiple method calls. Finally, Section V experimentally assesses the overhead incurred by this approach. Surprisingly, it shows that a richer language coupled with extended access to object properties, can in some cases simplify and speed up the monitoring process compared to a simpler, but less expressive solution. II. M OTIVATION AND R ELATED W ORK We first devise a simple scenario that describes important features of runtime monitoring. We then briefly review existing monitoring solutions and highlight the aspects of the problem that they do not cover appropriately.
an abstraction. In contrast, runtime monitoring operates on the actual implementation of a program.
A. A Simple Example Consider a simple class called Bank offering public methods with the following signatures:
B. Current Solutions The constraints in the previous example are characteristic in two respects. First, they involve sequences of method calls and argument values inside these method calls. Second, the sequence and values are interdependent: hence constraint S1 stipulates that the occurrence of close(x) prevents a call to any other method afterwards, but only for that same account number x. Similarly, S2 requires that methods isApproved(x,y) and the subsequent withdraw(x,y) have the same value y, but only if they also have the same value x. To enforce these properties at runtime, we first survey available tools and approaches. It shall first be remarked that many monitoring tools mostly gather statistics about method calls, such as execution times and hits, but do not track method call ordering. Examples of such tools include JAMon1 or MessAdmin2 . In that context, the term “monitoring” rather refers to performance monitoring. Closer to the task at hand is a framework called Java-MaC [11]. It instruments a Java program using its own routines in order to intercept the relevant method calls and the changes to monitored program variables. Monitors are programmed in a language called MEDL, a high-level scripting framework. However, although MEDL provides facilities for writing monitors, it does not generate them automatically from a declarative specification such as properties S1 and S2 above. Tracematches [1] and Java-MOP [14] are two frameworks that rely on AspectJ to instrument the relevant method calls in the program. This time, the properties to monitor are expressed in a declarative way, by simply describing a finitestate machine that specifies the valid orderings of method calls. In the case of Java-MOP, finite-state machines can also be replaced by other input languages such as Linear Temporal Logic or regular expressions. Finally, in the LIME runtime monitoring tool [9], specifications are written as annotations to Java interfaces or classes. The language provides constructs for referring to the instance of the object being called, the arguments passed to a method and its return value.
void open(int act) void close(int act) void withdraw(int act, int amt) boolean isApproved(int act, int amt)
A client can open a transaction with the bank for a particular account number; it can then withdraw money from any of the currently opened accounts, and close any of these accounts. An additional method, called isApproved, makes sure that a projected withdrawal receives the agreement of the bank before it is enacted. One can imagine an implementation of a Teller program that simply relays user commands as method calls to the bank. The teller keeps in memory the current account number and the amount of the last transaction. It allows a user to open multiple accounts and switch between them, but does not keep track of which accounts are open or which transactions are pending. Yet, we can reasonably impose a number of constraints on the way the bank’s four methods can be invoked. For example: S1. No operation on an account can be made after close S2. For the same account, the amount cannot change between an approval request and the following call to withdraw Obviously, using this simple teller can create problems with the Bank’s contract. For example, a user can open account 123, close it, and then attempt to withdraw $500 from it, violating constraint S1. Alternatively, a user can open account 123, ask for approval of $2,000, and then try to withdraw $3,000 instead, violating constraint S2. As a safety net, it may be desirable to enforce the previous constraints on the actual execution of a program, a process called runtime monitoring. This can be done for a number of reasons. For example, the satisfaction of requirements may sometimes depend on assumptions on the program that cannot be verified prior to the actual implementation of the system. In the present case, the Teller program depends on user commands that cannot be known until the program is actually run. Runtime monitoring can be hence used as a surrogate when the exhaustive, static verification of a program is not possible. Even in cases where static verification is possible in principle, it may be intractable in practice, for example due to the sheer size of the program’s state space. The issue becomes acutely relevant in the present scenario, where the properties to check involve a potentially unbounded number of different account numbers and withdrawal amounts. To this end, static verification seldom works with the actual implementation of a system, but rather on a formal model that must abstract away important details. Arguments in method calls are generally one of the first casualties of such
C. Drawbacks All these solutions, however, present issues when the properties to monitor deal with arguments occurring inside and across method calls. Classical finite-state machines, temporal logic or regular expressions cannot express such dependencies directly. Hence, in both Java-MOP and LARVA, dealing with parameters inside method calls involves dynamically instantiating monitors with arguments observed through monitored method calls. For example, one can devise a 1 http://jamonapi.sourceforge.net/ 2 http://messadmin.sourceforge.net
2
finite-state machine called S1(Account c) such that one new instance of S1 is created for every account c that appears during the execution. However, while instantiation is appropriate for expressing universal properties (which must be true for every instance of some class), they cannot be used to write existential properties: there is no way to tell whether a monitor instance should be enforced for some or all possible objects. Moreover, the specification of the allowed sequences of method calls becomes tangled with monitor instances. Hence in LARVA, a property that must apply to all accounts and all amounts will be specified by first defining a top level monitor, parameterized over some account act, specifying method calls to watch according to a finite-state machine. This monitor, in turn, will contain nested instances of another monitor, this time parameterized over the amount amt, and themselves watching another set of method calls. The constraints on method calls is the combined action of the nested monitors’ finite-state machines. This arguably makes the properties difficult to write. For example, to formalize even a simple property like S2, one must write a specification that deals at the same time with method calls caught by two levels of nested monitors, and parameterize the way they are instantiated. Hence a fair deal of the specification is actually a hand-coding of the monitor itself; moreover the specification of the method calls watched by the monitor, and the sequencing constraints on those calls, becomes scattered across various monitor declarations.3 Finally, each monitor instance is parameterized with an instance of a native program object. In the case where some properties to monitor require information about the state of the object, the monitor must be told to directly access the object. This also makes it very hard to verify a property on the trace once the program execution is over, as the monitor requires direct access to live program objects that no longer exist.
simple structure of attribute-value pairs. Hence the method call b.withdraw(123, 500) can be represented as the following message: withdraw 123 500
We choose XML markup as a convenient notation for method calls. This choice is sensible, as many applications, such as the Microsoft Event Viewer, export logs as sequences of XML-formatted elements for each event. One can see how the method name is enclosed in tags, while the value of each argument is contained within tags corresponding to the name of the respective argument, as declared in the method’s prototype. For the purpose of monitoring properties, an execution of a program can be reduced to a trace m of such messages, noted m = m1 , m2 , . . . . Data inside a message m can be fetched using a function, m(π), where π is the “path” inside the message leading to the set of values to obtain. For example, in the previous message, m(call/act) = {123}. Although each argument in a method call has a different name, nothing in the present framework prevents a message from having multiple element of the same name. B. A First-Order Temporal Logic: LTL-FO+ We have seen in Section II-B how the input languages of existing tools are not rich enough to take data-aware constraints into account. A suitable solution must offer both a way to access method parameters, and a monitoring algorithm that can record these values and recall them at a later time for comparison. The language we present in this section is LTL-FO+ , a first-order extension of a well-known logic called Linear Temporal Logic (LTL) [15], which has been shown to be appropriate for the modelling of data-aware properties. LTL-FO+ is not the focus of the present paper; the reader is referred to [8] for an in-depth coverage. The building blocks for asserting properties over message traces are atomic propositions, which are of the form x = y, where x and y are either variables or constants. These atomic propositions can then be combined with Boolean operators ∧ (“and”), ∨ (“or”), ¬ (“not”) and → (“implies”), following their classical meaning. In addition, LTL temporal operators can be used. The temporal operator G means “globally”. For example, the formula G ϕ means that formula ϕ is true in every message of the trace, starting from the current message. The operator F means “eventually”; the formula F ϕ is true if ϕ holds for some future message of the trace. The operator X means “next”; it is true whenever ϕ holds in the next message of the trace. Finally, the U operator means “until”; the formula ϕ U ψ is true if ϕ holds for all messages until some message satisfies ψ. Finally, LTL-FO+ adds quantifiers that refer to parameter values inside messages. Formally, the expression ∃π/x : ϕ(x)
III. F IRST-O RDER RUNTIME M ONITORING The properties in the previous scenario have been dubbed “data-aware” constraints in previous work. This section shows a runtime monitoring framework for data-aware constraints that addresses the aforementioned issues. A. Method Calls as Messages Method calls can be likened to a form of “event”. In the present case, these events correspond to the method calls relevant to the properties to enforce, along with all the information required to properly evaluate the constraints. For each method call, one must therefore record the name of the method, as well as the name of each argument and its actual value in that particular method call. This data bundle, which we will also call a message, can be represented by a 3 An example of nested finite-state machines can be found in the LARVA User Manual [5, p. 18].
3
PROTOTYPES void Bank.open(int act); void Bank.close(int ac); void Bank.withdraw(int act, int amount); boolean Bank.isApproved(int act, int amt);
Pointcut Event formatter
m
Advice execution
Outcome
Monitor
Program
Figure 1.
SPEC LTL-FO+ G ((call/method/close) -> ([a1 call/act] (X (G ([a2 call/act] (!((a1) = (a2))))))))
Aspect
Runtime monitoring architecture
@FALSE { System.out.println("This is forbidden"); }
states that in the current message m, there exists a value v ∈ m(π) such that ϕ(v) is true. Dually, the expression ∀π/x : ϕ(x) requires that ϕ(v) holds for all v ∈ m(π). When the context is clear, we abbreviate ∃π/x : x = k as π/k, stating that the (only) value at the end of path π is k. Equipped with this logic, constraints from the bank example can be formalized. For example, constraint S1 becomes:
Figure 2.
•
G (call/method/close → ∀ call/act/a : X G (∀ call/act/a0 : a 6= a0 )) This formula states that whenever the close method is called, then for every account number a inside that message, we have that any future message with an account number i0 is different from i. This is indeed equivalent to S1, which forbids any operation involving an account once it has been closed. Property S2 can be formalized in a similar way.
•
Monitor specification for property S1.
arguments to be named; the parameters don’t need to have the same name as their declaration in the original source code. The second section, SPEC LTL-FO+, provides the LTL-FO+ formula to monitor. The formula shown in the figure is a plain-text equivalent of the formula we have seen earlier. It will be automatically parsed by the runtime monitor later on. The last section, @FALSE, is optional code that is to be executed whenever the property is violated (a similar section, @TRUE, can specify what to do if the property is satisfied).
Once the input specification is written, a PHP script (not shown in Figure 1) transforms the input specification into an executable AspectJ file called the Messenger. This file can then be included to the original Java program like any other AspectJ file. The Messenger is responsible for declaring appropriate pointcuts to intercept the relevant method calls. When instantiated, it also creates an instance of the BeepBeepMonitor. A separate .jar file, contains the code for the BeepBeepMonitor and must be linked to the original project.5 During the execution of the program, each event matching one of the pointcuts is caught by the Messenger and goes through an Event formatter (cf. Figure 1). The formatter gathers the information about the method call and generates the corresponding XML message as described earlier. The names given to each argument are those provided in the prototype declaration for this particular signature. This message is then relayed to the monitor for verification. Finally, the Messenger retrieves to the monitor’s outcome. According to the response, the Messenger can execute the recovery code given in the input specification. The described framework is modular, in the sense that the runtime monitor can be substituted by any other algorithm. Indeed, the event formatter merely produces a standard XML representation of method calls. Any runtime monitor accept-
C. Monitoring Framework An algorithm for the runtime monitoring of LTL-FO+ formulæ has already been presented in the context of web services and implemented as a stand-alone library called the BeepBeepMonitor.4 The proposed architecture, described in Figure 1, leverages this implementation to provide a modular runtime monitoring framework. In its current incarnation, our architecture allows one to write and monitor specifications in LTL-FO+ , and uses the AspectJ library to trap relevant method calls. Although the examples we have shown are based on Java, the solutions we put forward are not Java-specific, with the exception of the event formatter (for which adaptations to other languages can easily be written). First, a user writes an input specification describing the method calls to intercept and the property to monitor on that sequence of calls. The format intentionally resembles a similar input language for plugins in the Java-MOP framework. Figure 2 shows the input file required for the monitoring of property S1: • The first section, called PROTOTYPES, declares the signature of all the methods to be caught, including the class name they belong to. The prototypes allow method 4 The description of this algorithm is not the focus of the present paper. The reader is referred to [8] for a complete description and a proof of soundness.
5 The PHP script and complete source code for the monitor can be used online at http://beepbeep.sourceforge.net/java-monitor/monitor-plugin.php
4
AdvancedBank
bank does not manage, i.e. that has not been passed to the bank through its open() method: A1. To withdraw money from an account, the Bank must be managing it. As a second example, consider the following scenario. After opening an account (say 123), a user requests an authorization for a withdrawal of $1,000. The user then finds a way to modify the amount in the authorization and increases it to $2,000. He then calls withdraw() using this tampered authorization. This calls for a second constraint: A2. The amount of an authorization passed to withdraw() must be the same as when the authorization was granted. The monitoring of these two constraints is problematic for two main reasons. First, property A1 correlates the same account in two method calls, but in reverse: an account can be passed to withdraw() only if it has been seen in a call to open() in the past. There is no way to express that fact directly with LTL-FO+ . It is tempting to write the formula as “for every account x and amount y, there is no withdraw(x,y) until an open(x)”. However, quantifiers in LTL-FO+ must always refer to values occurring in the current message of the trace; one cannot say “for all values” taken from a general domain. It hence looks like the specification language is insufficient to cover this case.6 Second, although the properties track arguments across multiple method calls, as in the first example, this time the arguments are instances of classes Account and Authorization. While the event formatter can easily put the value of a primitive type inside an XML message, non primitive data types don’t have such a well-defined “value”. Their internal state can be formed of any number of member fields and properties; not all of them are relevant for the property to monitor.
void withdraw(Account a, Authorization auth) void open(Account a) boolean manages(Account a) Authorization getAuthorization(Account a, int amount) Authorization
int getId(); int getAmount(); Account
void setBalance(int balance); int getBalance(); int getNo(); Table I P ROTOTYPES FOR CLASSES OF THE A DVANCED BANK EXAMPLE
ing events in this message format, and returning an outcome in a standardized way, can replace the BeepBeepMonitor currently in use. For example, if the properties to monitor only necessitate a simple finite-state machine, or a context-free grammar, a different monitor could be instantiated without replacing the rest of the framework. IV. N ON - PRIMITIVE DATA T YPES As we have seen, the event formatter creates an XML “document” from a method call; an interesting consequence of this design is that, while the PROTOTYPES section is language-specific, the LTL-FO+ specification is not: it is expressed on sequences of XML messages produced by the event formatter. Therefore, by providing various sets of instructions to the event formatter, differing languages (and even differing implementations) can all be projected onto message sequences of the same format, which effectively shields the temporal specification and its related monitor from implementation-dependent considerations. We shall see in this section that this shielding mechanism becomes even more important when the properties to monitor consider program objects of non-primitive types.
B. Querying Object Fields and Methods One shall remark that these two problems are solved if we allow objects’ fields and methods to be queried at runtime. For constraint A1, it suffices to notice that the bank keeps track of the accounts it has opened, and that this can be asked through the manages() method. Constraint A2 can likewise be enforced by querying and comparing the returned values of getAmount() and getId() fields of authorizations passed to getAuthorization() and withdraw(). We designed our runtime monitoring platform so that arguments in method calls can be queried. This can be stipulated in the PROTOTYPES section of the specification, as is shown below:
A. The Bank Revisited We consider a refined version of the bank example, made of three classes whose signatures are given in in Table I. This time, an account is not represented by a simple number, but rather by a class Account carrying a number and a balance. The AdvancedBank’s open() method is passed instances of this class. In addition, the bank now replies to authorization requests by returning an instance of class Authorization. An authorization contains the amount for which the withdrawal was requested, as well a randomly generated authorization number. This authorization must be passed to the bank’s withdraw() method afterwards. A first natural constraint we can express on this scenario is that one cannot withdraw money from an account that the
PROTOTYPES void AdvancedBank.withdraw(Account a, int amt, Authorization au); 6 Note that the same problem occurs if we try to monitor a similar property on the simple bank example.
5
{amount}
withdraw {this.manages(a)}
The LTL-FO+ formula that monitors this property is:
This time, the prototype declaration is followed by an XML template. This template tells the formatter what XML message to create upon catching a call to the method matching the prototype. This message is free form; one can use arbitrary element names and whatever structure is deemed appropriate, including nested elements if necessary. Everything that does not appear between braces is copied verbatim. The portion between braces, however, is Java code. When creating the XML structure upon a method call, the formatter replaces it by the result of evaluating the enclosed expression. This expression can involve any of the arguments from the method call; in addition, the instance of the object that has been called can also be referred to using the keyword this. In the previous example, the prototype declaration instructs the formatter to create a simple XML message with a method element containing the value withdraw, followed by an element is-managed. The contents inside the latter should be computed by calling the method manages() of the current instance of AdvancedBank (this), with the Account instance of the current call to withdraw (a). In other words, every time the withdraw() method will be called, the formatter will send to the monitor a message that indicates whether the account being withdrawn from is indeed managed by the bank at that precise moment. Remark how the structure does not include all the arguments of the call: nothing about amt and auth is sent to the monitor. Not much is sent about the account either: only the fact that the bank manages it or not. Hence information about irrelevant arguments is not unnecessarily written and processed by the monitor. Monitoring constraint A1, based on that message structure, becomes straightforward. The LTL-FO+ formula is simply:
G ∀call/method/m : (m = withdraw → ∀call/balance/b : ∀call/amount/a : a < b) C. Complex expressions One can see how this principle allows one not only to monitor primitive types passed in method calls, but also any expression on the arguments that returns primitive types. Monitoring constraint A2 requires correlating the id and amount member properties of the Authorization returned by getAuthorization() with the same values in the Authorization passed to withdraw later on. This can be done by declaring XML structures for two method prototypes as follows: PROTOTYPES void AdvancedBank.getAuthorization(Account a, int amount); getAuthorization {return.getId()} {return.getAmount()} void AdvancedBank.withdraw(Account a, Authorization auth); withdraw {auth.getId()} {auth.getAmount()}
The first XML structure begins with a element instead of a . This tells the event formatter to produce a message once the method call is finished, rather than before is is executed. This way, one can express properties on the object returned by the method, which is accessed through the keyword return. Hence, return.getId() amounts to calling the getId() method on the returned object. The second XML message traps calls to withdraw. This time, the created message includes the value of the id and amount fields of the authorization passed to the method. This is done by calling getId() and getAmount() on that authorization, and placing the values inside the respective elements. The LTL-FO+ formula that monitors this constraint can then be written as:
G (call/method/withdraw → call/account/is-managed/true) Intuitively, the monitor watching this property will check that, every time a call b.withdraw(a, amt, au) is called on an instance b of AdvancedBank, the result of b.manages(a) at that precise moment returns the boolean true. Similarly, we could check, for example, that the balance of the account contains at least the amount to be withdrawn. It would suffice to create an XML structure recording the account’s balance and value of the amount argument, as follows:
G (return/method/getAuthorization →
PROTOTYPES void AdvancedBank.withdraw(Account a, int amount, Authorization auth); withdraw {a.getBalance()}
∀ return/id/i : ∀ return/amount/a : X G (call/method/withdraw → ∀ call/id/i0 : ∀ call/amount/i0 : (i = i0 → a = a0 ))) The formula stipulates that, every time a getAuthorization() returns an authorization 6
whose ID is i and amount is a, then any future call to withdraw() is such that the authorization auth passed to it, if it has ID i, has amount a. This, in effect, prevents passing to withdraw() an authorization whose amount has been changed after its creation. Note how, although the method calls involve non-primitive objects (accounts and authorizations), neither actually makes its way to the monitor. The event formatter rather sends primitive results computed over these objects. In a way, the event trace produced by the event formatter can be seen as a simplified projection of the global program state, where only relevant (and simple) fields and properties about each object are kept in each trace message. Hence storing the trace for later use does not require storing snapshots of complete objects at each moment, which would be a considerable undertaking.
Number of traces
60 50 40 30 20 10 0 [-.08, -.04)
[-.04,[-.02, [0, -.02) 0) .02)
[.02,.08)
[-.06,-.03) [-.03,0) [0,.02) [.02, .04)
A1
[.04,.08)
A2 Overhead per method call (ms)
(a) Simple Bank
Number of traces
60
V. E XPERIMENTS AND D ISCUSSION
50 40 30 20 10 0 [-.08, -.04)
In order to assess the feasibility of the approach, we conducted a set of experiments on the examples described in the previous sections. In this section, we describe and discuss these results.
[-.04,[-.02, [0, -.02) 0) .02)
[.02,.08)
[-.06,-.03) [-.03,0) [0,.02) [.02, .04)
A1
[.04,.08)
A2 Overhead per method call (ms)
(b) Advanced Bank
A. Evaluation of Overhead
Figure 3. Distribution of overhead per method call (in milliseconds) for each property in the two bank scenarios.
For both the Simple Bank and the Advanced Bank scenarios, a main Java program produced 100 method calls to the bank. The calls were chosen at random depending on the current state of the opened accounts. For example, if account number c was opened, the action “close account c” was added to the pool of possible future method calls; if an authorization for account c and amount a was requested, the action “withdraw a from c” was added to the pool of possible future method calls, an so on. The end result is a script performing a large number of authorization requests, withdrawals, account closings and openings. The script’s non-determinism was managed by the use of a single random number generator using a parameterized start seed; hence for the same seed, the script generated the same sequence of method calls with the same arguments. In the following experiments, the script was run with 100 different seed values, effectively generating 100 different programs interacting with the bank. In total, the monitor processed roughly 600,000 method calls on four different properties. Runtime Overhead. The first experiment involved the measurement of the runtime overhead incurred by the use of the monitoring platform. Hence, we compared the running times of each of the 100 programs, both with and without the platform, and for each of the four constraints. The time difference per method call is plotted in Figure 3. The results show an overhead ranging between 1 and 100 ms per method call. In some cases, such as properties A1-A2, the running times are so close to the system clock’s precision that the difference sometimes appears as negative. One can see that the overhead is proportional to the complexity of
the formula to monitor, with S2 being the most complex formula of all the examples described in the paper. This led us to suspect that a large part of this overhead was in fact contributed by the external runtime monitor which is an external JAR file that can be replaced by a different component. We hence measured the elapsed time spent inside the sole event formatter; the overhead contributed by the XML event formatter itself accounts for only 5-10 microseconds per method call, and this is irrespective of the property to monitor. Based on these positive results, we shifted our focus to realworld code and ran similar tests on the DaCapo Benchmark [3] (release 9.12), which consists of a set of open source, real world applications with non-trivial memory loads. The fourteen applications range from transformation of XML documents and production of Scalable Vector Graphics to microcontroller simulation and static code analysis. For each of the 14 test suites included in the benchmark, we monitored the following property: a call to next() on an instance of Iterator must only be made when there are still elements to iterate over. Each test suite was run 10 times; the average overhead per method call is shown in Table II. The average overhead for all benchmarks is 5.06 ms per method call. The property is of similar complexity as property S1 and presents commensurate overhead. One can hence expect that the measured overhead in our synthetic tests extends to real7
Overhead (ms) 14.95 0.69 23.97 0.70 2.34 -2.24 0.80
Test suite lusearch pmd sunflow tomcat tradebeans tradesoap xalan
Overhead (ms) 13.75 0.44 -0.95 0.99 2.11 -5.45 18.79
10
6
10
Time (ms)
Test suite avrora batik eclipse fop h2 jython luindex
10
5
4
10
Table II AVERAGE OVERHEAD PER METHOD CALL FOR THE DAC APO B ENCHMARK .
10
3
2
0
10
20
30
40
50
Number of accounts
world programs as well. Object Querying vs. Independent Monitoring. The second experiment aimed to assess the impact of querying properties of arguments in method calls. To this end, we considered the property “a call to withdraw() is forbidden once an account has been closed” on the Advanced Bank example. There are two ways of monitoring this property: 1) A first way is to track calls to open and close, maintain in memory the list of accounts that are closed and raise an error whenever one of these account numbers appears in a call to withdraw later on. This yields a formula that uses first-order quantifiers to correlate account numbers appearing in calls to close, withdraw and open, similar to the specification in Figure 2. 2) A second way is to simply query the bank whenever withdraw is called and check whether the account under consideration is indeed managed by the bank —this is exactly property A1. While both methods return equivalent results, the first one (the “autonomous” approach) has the monitor deduce the bank’s list of open accounts by keeping track of arguments occurring in calls to open() and close(), while the second one (the “query” approach) directly queries that state on the bank itself at runtime using method manages(). We ran 100 instances of the program described in the previous experiment, this time fixing the number of method calls to 1,000 and varying the maximum number of open accounts between 1 and 50. Figure 4 plots the running times for each program with respect to the number of accounts, for both the autonomous and the query approach. The measured running times clearly favour the query approach, and especially more so as the number of accounts handled by a program increases. This is true even if the processing of each event is heavier and involves calling extraneous methods on the program’s objects. This shows how, in some cases, the pre-processing of an event before passing it to the monitor can end up simplifying the monitoring task. Table III shows statistics that can explain that behaviour First, for each method call, the query approach requires twice the time in the event processor, compared to the autonomous approach. However, this is largely offset by the fact that in the query approach, only withdraw() needs to be handled.
Figure 4. Scatterplot of running time for the same programs, monitored with the “autonomous” (×) and the “query” approach (+).
Processed calls Formatting time per call (µs) Total formatting time (ms) Total monitoring time (s)
Autonomous 100,000 4.24 279 835.5
Query 30,980 9.01 424 34.8
Table III S TATISTICS ABOUT THE EVENT FORMATTER , USING THE AUTONOMOUS AND THE QUERY APPROACH .
In the autonomous approach, methods open(), close() and withdraw() must be instrumented. Hence the query approach processes less than a third of the events compared to the autonomous approach. Second, in the autonomous approach, the monitor must memorize the numbers of closed accounts (through its processing of the close() method), and check upon calls to withdraw() whether the current account number belongs to that list. In the query approach, the property to verify does not require to persist any information between method calls. Hence the task performed by the monitor is much simpler, and is handled roughly 50 times faster. B. Field Access vs. Monitoring Complexity The second experiment can be viewed as an illustration that the access to an object’s member fields and the use of first-order quantifiers in specifications actually complement each other. In the autonomous approach, only the account numbers appearing in method calls are processed by the monitor: the bank’s internal state is never queried. Therefore, to record which accounts are open and which are closed, we need to use a rich specification language with LTL temporal operators and first-order quantifiers. The quantifiers are used to fetch an account number x in a call to close(), and to check whether subsequent calls to withdraw() use that same x. On the opposite, the query approach relies on the bank’s internal state, through its manages() method. This ef8
fectively outsources the tracking of account numbers and spares the monitor from managing this information by itself. Consequently, the property to monitor becomes a simple, propositional logic formula that merely evaluates a condition on the state information that was fetched by the event formatter. The presence into the same monitoring platform of both a rich input language and the possibility to query objects’ states provides flexibility in situations where the same property can be evaluated by different, but ultimately equivalent means as in the example above. When the information is available and can be trusted, relying on the object’s own state can speed up monitoring and greatly reduce the complexity of the property to verify. Otherwise, the presence of first-order quantifiers in the specification language makes it possible to deduce whatever state information is necessary through the use of correlations between values across multiple method calls.
However, this approach creates an important problem. All properties shown in this paper require some arguments’ properties to be saved, and compared with other properties later in the execution. If events contain references to objects, then these objects must be memorized and recalled. But since Java objects are always passed by reference, this becomes impossible as soon as the state of the object changes between the moment it is “captured” by the monitor and the moment it is queried. The only workaround is to perform a deep copy (clone) of the said object, and to refer to that clone. Yet, not all objects are cloneable (consider a FileInputStream), and even those that do can require the duplication of a considerable portion of the program’s state. It quickly becomes clear that this solution is much more complex. In addition to being impractical in most cases, the solution is also overkill: in most cases, only a small facet of the object is required to monitor a property (the authorization id and amount in our scenario), even if the object itself contains many more member fields. Using an event formatter produces a projection of the program’s state that contains only the information required. Finally, since each processed method call created by the event formatter is standard XML, the execution trace can easily be recorded on disk and processed post mortem, even by a different algorithm, without losing any of the relevant information that was available at runtime.
C. Purpose of the Event Formatter The execution of Java code prior to sending the data to the monitor might seem odd. Indeed, arbitrarily complex blocks of code, executed each time an event is processed, could cause unwanted side effects and incur unreasonable overhead. However, when used carefully, this mechanism offers appeal in two respects. Independent Event Format. First, it decouples event catching from property monitoring, making it an explicit twostage process. The interception of events and their formatting into XML isolates the monitor from any language-dependent considerations. The monitor receives a “sanitized” snapshot of whatever program state information is necessary to evaluate a property. Therefore, the monitor need not be concerned with the specifics of the program to monitor; as a matter of fact, the monitor used in this paper is actually a direct reuse of a component initially created to validate web applications in previous work [7]. Any processor that accepts XML documents as its event format can be plugged in place of this monitor. In addition, the same property can be monitored on two programs whose method names and arguments are different. Specifications for both programs can be written in such a way that the event formatter ultimately produces similar messages in both cases, despite the fact the information they contain had to be fetched in a different way. No Native Objects Inside Events. To allow access to objects’ fields and methods, an alternate possibility would have been to pass the objects directly to the monitor. In such a case, the message sent to the monitor is no longer a string of XML text, since the “value” inside some elements must be a reference to a Java object in the program to monitor. This is possible, since the current version of our runtime monitor is also implemented in Java and could ultimately be able to manipulate references to other Java objects.
VI. C ONCLUSION The paper presented a runtime monitoring platform for programs where method calls are trapped and processed using an aspect-oriented library. Its modular architecture has at its core an event formatter that converts information about method calls into a flexible, XML-like “message” that can then be sent to an external runtime monitor class. The formatter can be directed to query the arguments involved in a method call, and record the resulting values into the message. Coupled with a rich specification language called LTLFO+ , this platform allows us to monitor complex properties that correlate the state of objects’ member fields across multiple method calls —these have been dubbed “data-aware” temporal properties. This project lends itself to multiple extensions. First, as long as an event formatter can be designed for some other programming languages, such as Python, then our runtime monitoring algorithm works readily with the XML-formatted events —something that the other, Java-centric solutions in the literature cannot do. Moreover, in user interfaces built using the GUI framerowks, many window events are processed through a single entry point, the event listener, and are only distinguished through the various Event objects that are passed to this listener. Properties on the consistency of the user interface hence must track the state of the same component across multiple calls to the event listener. In 9
addition, the observation that part of the monitor’s task can be simplified by querying the object’s own internal state opens the way to generic optimization techniques that could reduce runtime overhead.
[12] D. Lee, D. Chen, R. Hao, R. E. Miller, J. Wu, and X. Yin. A formal approach for passive testing of protocol data portions. In ICNP, pages 122–131. IEEE Computer Society, 2002. [13] M. Leucker and C. Schallhart. A brief account of runtime verification. J. Log. Algebr. Program., 78(5):293–303, 2009.
R EFERENCES [1] P. Avgustinov, J. Tibble, and O. de Moor. Making trace monitors feasible. In R. P. Gabriel, D. F. Bacon, C. V. Lopes, and G. L. S. Jr., editors, OOPSLA, pages 589–608. ACM, 2007.
[14] P. O. Meredith, D. Jin, F. Chen, and G. Rosu. Efficient monitoring of parametric context-free patterns. Autom. Softw. Eng., 17(2):149–180, 2010. [15] A. Pnueli. The temporal logic of programs. In FOCS, pages 46–57. IEEE, 1977.
[2] E. Bayse, A. R. Cavalli, M. Núñez, and F. Zaïdi. A passive testing approach based on invariants: application to the WAP. Computer Networks, 48(2):235–245, 2005.
[16] H. Ural and Z. Xu. An EFSM-based passive fault detection approach. In A. Petrenko, M. Veanes, J. Tretmans, and W. Grieskamp, editors, TestCom/FATES, volume 4581 of Lecture Notes in Computer Science, pages 335–350. Springer, 2007.
[3] S. M. Blackburn, R. Garner, C. Hoffman, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovi´c, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In OOPSLA ’06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-Oriented Programing, Systems, Languages, and Applications, pages 169–190, New York, NY, USA, Oct. 2006. ACM Press. [4] F. Chen, M. D’Amorim, and G. Rosu. Checking and correcting behaviors of Java programs at runtime with Java-MOP. In Workshop on Runtime Verification (RV’05), volume 144(4) of ENTCS, pages 3–20, 2005. [5] C. Colombo. LARVA parser user manual, 2008. http://www. cs.um.edu.mt/svrg/Tools/LARVA/LARVA-manual.pdf. [6] C. Colombo, G. J. Pace, and G. Schneider. Dynamic event-based runtime monitoring of real-time and contextual properties. In Formal Methods for Industrial Critical Systems (FMICS), volume 5596 of Lecture Notes in Computer Science, pages 135–149, L’Aquila, Italy, 2008. [7] S. Hallé and R. Villemaire. Browser-based enforcement of interface contracts in web applications with BeepBeep. In A. Bouajjani and O. Maler, editors, CAV, volume 5643 of Lecture Notes in Computer Science, pages 648–653. Springer, 2009. [8] S. Hallé and R. Villemaire. Runtime enforcement of web service message contracts with data. IEEE Trans. on Services Computing, 2011. DOI: 10.1109/TSC.2011.10. [9] K. Kähkönen, J. Lampinen, K. Heljanko, and I. Niemelä. The LIME interface specification language and runtime monitoring tool. In S. Bensalem and D. Peled, editors, RV, volume 5779 of Lecture Notes in Computer Science, pages 93–100. Springer, 2009. [10] G. Kiczales, E. Hilsdale, J. Hugunin, M. Kersten, J. Palm, and W. G. Griswold. An overview of AspectJ. In J. L. Knudsen, editor, ECOOP, volume 2072 of Lecture Notes in Computer Science, pages 327–353. Springer, 2001. [11] M. Kim, M. Viswanathan, S. Kannan, I. Lee, and O. Sokolsky. Java-MaC: A run-time assurance approach for Java programs. Formal Methods in System Design, 24(2):129–155, 2004.
10
