3. RMIX Asynchronous Invocation Model. Several design goals were targeted during the design of ... Another category of calls, described as one way calls, is.
Semantic Aspects of Asynchronous RMI: the RMIX Approach Dawid Kurzyniec, Vaidy Sunderam Emory University, Atlanta, USA {dawidk,vss}@mathcs.emory.edu Abstract Remote Method Invocation is one of the most popular communication paradigms in distributed computing. However, its synchronous nature may affect application performance. Asynchronous, or non-blocking, RMI addresses the issue by implicitly introducing concurrency into an application and allowing it to interleave computations with communications. In this paper, we analyze certain semantic issues raised by asynchronous RMI, such as execution order, exception handling, cancellation, data consistency, and more. Solutions are proposed, and a realization of asynchronous RMI within our multi-protocol, extensible communication framework for Java (termed RMIX), is presented.
1. Introduction Remote Method Invocation – an object-oriented variant of Remote Procedure Calls – is one of the most popular communication paradigms currently used in the mainstream of distributed computing, both in the industrial and scientific domains [14, 9, 13, 4]. One of the key factors that makes RMI so appealing is that it hides the complexity of remote communication behind traditional, well-understood semantics of method calls. Not only does it simplify the development of distributed applications, but it also minimizes the changes needed to distribute legacy applications. However, it has been shown [10, 15] that the synchronous nature of RMI (as contrasted, for example, with asynchronous message passing) may affect overall application performance. This is due to the fact that the invoker of a remote method is always blocked until the method completes. Even in cases when execution costs are negligible, overall performance is bound by network latency which may be significant especially in geographically distributed applications. One common solution to the above is to explicitly introduce concurrency, allowing the application to interleave computations with communications by using multiple threads of execution. However, such an approach may require significant changes to the application code and is not always feasible. A complementary and less invasive ap-
proach is to make RMI calls non-blocking, or asynchronous, thus introducing concurrency implicitly. Seemingly trivial, it brings about several interesting semantic consequences (related to execution order, exception handling, and cancellation, to name a few). There have been several proposals to introduce asynchronous RMI in Java [10, 15, 2, 6]. However, the majority of these mechanisms focused on performance aspects, and tended to neglect the semantic issues that arise. Notwithstanding that high performance is the ultimate goal of asynchronous RMI, the semantic factor is inarguably crucial for achieving correctness and determinism. In this paper, we analyze asynchronous RMI from the semantic perspective, and describe concrete software solutions which have been implemented within the multi-protocol, general-purpose communication framework for Java called RMIX [7, 11]. Noteworthy, although this paper focuses mostly on Java RMI and the extensions of the RMIX framework, the discussed issues and their suggested solutions are relevant to RMI as a communication paradigm in general.
2. Overview of RMIX With the advent of grid computing, peer-to-peer, and Web Services, heterogeneous distributed computing has received renewed attention. Java libraries supporting a variety of communication fabrics do exist, but they require application developers to adjust to certain APIs, so generally they are not interoperable. The RMIX project aims to organize such independent solutions into a common, provider based framework founded on the RMI paradigm. RMIX specifies a facade client API through which RMI functionality can be accessed regardless of the underlying implementation and protocols, and a provider side API to be implemented by service providers. These concrete provider implementations can then be dynamically plugged into the framework, thus supplying communication capabilities. To date, supported protocols include JRMPX [14], ONC-RPC [5], and SOAP [3]. Additionally, RMIX features several general-purpose enhancements over Java RMI, including dynamic stubs, invocation interceptors, and SSL support. RMIX has become a mature system, and it is publicly available [11].
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
The ideas suggested in this paper have been realized in RMIX in terms of APIs and behavioral specifications which are guaranteed for applications and which must be adhered to by RMIX service providers. The concrete implementations are briefly outlined in Section 4. However, it is important to emphasize that the framework is not limited to those implementations, as additional ones may be furnished by independent suppliers.
3. RMIX Asynchronous Invocation Model Several design goals were targeted during the design of the RMI asynchrony model discussed in this paper. One of the most important was to maintain server-side transparency at the application level, i.e. making server objects oblivious to the fact that their methods may be invoked asynchronously. Moreover, support for asynchrony has been made optional, so that service providers are not forced to implement it and applications are not forced to use it. Other goals included: retaining interface simplicity, enabling high performance, and avoiding unanticipated behavior (that is, to use defaults which are intuitively expected). The details are presented further in this section.
interface Hello extends Remote { void hello(String greeting) throws RemoteException; } interface AsyncHello extends Hello { Future asyncHello(String greeting) throws RemoteException; Future cbasyncHello(String greeting, Callback cb) throws RemoteException; void onewayHello(String greeting) throws RemoteException; }
Figure 1: Canonical example of asynchronous interface
interface Future { boolean isDone(); Object get() throws InterruptedException, ExecutionException; Object get(long timeout, TimeUnit granularity) throws InterruptedException, ExecutionException, TimeoutException; } interface Callback { void completed(Object result); void failed(Throwable cause); }
3.1. Syntax
Figure 2: “Future” and “Callback” interfaces
A distinguishing factor of the asynchronous RMI paradigm (in comparison to ordinary RMI) is that the result of an invocation is not available to the caller immediately after the method returns. Implementation must thus somehow provide the caller with the ability to retrieve that result when it arrives. The result itself may be one of: (1) a successfully returned value, (2) a mere indication of successful completion in case of methods returning “void”, and (3) an indication of failure with appropriate exception reporting. Canonical solutions [1] (sometimes used in combination) include: • futures. Asynchronous invocation returns a so-called “future” which may be used to query or poll for a result arrival; • completion callbacks. At the time of invocation, the caller registers a function to be invoked by the environment upon completion; • result queues. At the time of invocation, caller specifies a queue in which the result should be placed once it arrives. Another category of calls, described as one way calls, is characterized by offering no completion notification whatsoever. In some cases, one-way calls may be implemented more efficiently than asynchronous calls. RMI methods are invoked via client-side stubs. Each stub represents a specific server object and mirrors that object’s
remote interfaces. This approach makes the syntax of a remote call virtually identical to that of a local method invocation. Asynchronous RMI unavoidably complicates the syntax as it has to account for the aforementioned aspects of completion notification. 1 The approach proposed within the RMIX framework, which we believe retains RMI syntactical simplicity to the extent possible, is achieved via asynchronous interfaces. Client-side stubs implement these asynchronous interfaces in addition to the interfaces of a server object, thus allowing the client to invoke supplementary methods. RMIX interprets invocations of such methods as requests to make asynchronous (or one-way) calls, and the actual methods to be called on the server are resolved using a simple naming convention, as suggested in the example in Figure 1. As shown, up to three asynchronous methods are introduced per each original one: two methods for asynchronous calls with and without a callback, respectively, and an optional method for a one-way call (only if the original method returned “void”). The Future and Callback interfaces are illustrated in the Figure 2; the Future interface is based on the java.util.concurrent package anticipated in Java 1.5 [8]. Note that RMIX does not explicitly support result queues: this is because they may decrease the level of isolation between client threads, and moreover, their 1
retaining the syntax may sometimes be possible, but only in specific cases [2].
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
functionality may be easily implemented on top of completion callbacks if needed. Importantly, the features described above affect only the client side of the application, and only if desired. The server object needs to implement merely the original, synchronous interfaces. The stub dynamically created at the client side (e.g. during unmarshalling of a remote reference returned from a previous call) will implement asynchronous interfaces only if the application contains them in the class path, and only if asynchrony is supported by the underlying communication service provider, as determined at run time.
3.2. Data Consistency At the client side, remote call consists of three phases: (1) parameter marshalling, (2) awaiting response from the server, (3) unmarshalling the result. Naturally, an asynchronous call is expected to return control to the invoker before (2) and (3) take place and perform them in a background. However, it is less obvious which strategy is appropriate for parameter marshalling. Asynchronous processing introduces the risk of data inconsistency, illustrated in Figure 3. The issue may occur if the caller thread modifies parameters after passing them to a remote call but before they are marshalled by the background thread. (Importantly, there is no such risk in the special case when the parameters are immutable). On the other hand, synchronous marshalling may cause the call to be blocking, e.g. when the communication rate is high enough (or the message large enough) to fill up send buffers. The issue can be alleviated to some extent by employing additional buffering (e.g. making deep in-memory parameter copies), but it usually only defers the problem until the memory buffers fill up, too. In accordance with the design goals outlined previously, the solution proposed within the RMIX framework is to use a safe default; that is, communication providers are required to ensure parameter consistency. The exact realization is left to providers and may involve synchronous marshalling, deep copying, and immutability detection. This default can be overridden by the client on a per-stub basis to
allow early returns; in that case, the responsibility for ensuring data consistency is delegated to the application.
3.3. Ordering of Remote Calls A fundamental property of the synchronous RMI paradigm is that remote calls performed from a single thread are totally ordered (never executed concurrently) and their order of execution matches the order of invocation. On the other hand, there are no guarantees about calls made from distinct threads (at least in Java RMI) – they may execute concurrently, and even if the server side enforces ordering via synchronization, the lock acquisition order remains undeterministic. Asynchronous RMI can be thought of as a way to increase concurrency. Hence, in typical usage scenarios it is desirable to relax the ordering semantics of consecutive calls made from a single thread. For instance, if such asynchronous calls are made to remote objects on different hosts, it is usually expected that processing would begin independently at each host. On the other hand, there are situations where that order should be preserved. Consider an event notification example shown in the Figure 4, which represents a typical application of asynchronous RMI. Without ordering guarantees, the “completed” notification may precede, or be executed concurrently with, the “started” notification, leading to unexpected behavior. A possible resolution would be to leave the choice of an appropriate ordering strategy up to the user as a qualityof-service parameter. However, for the sake of simplicity, RMIX defines default ordering semantics that, as the authors believe, is appropriate in the vast majority of cases. That default is to guarantee that total order is preserved for calls performed from a single thread via the same stub. This void performAction() { remoteListener.asyncNotify("started"); doTheWork(); remoteListener.asyncNotify("completed"); }
:stub
:stub create()
:param
:target
“started”
asyncCall() modify()
“completed” read()
Figure 3: Possible data inconsistency related to asynchronous marshalling
Figure 4: Ordering violation in asynchronous event notification example
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
way, concurrency can be enabled even between calls to the same remote object (as long as they are invoked via distinct stubs), whereas the event-notification and message-passing applications may still rely on ordering guarantees. To preserve the order of calls, they must be queued and executed sequentially. One possible solution is to organize the queues at the client side, with per-queue threads processing calls in a synchronous manner. The important advantage of such an approach is that it does not affect the communication protocol or the server-side middleware, thus enabling implementations making asynchronous calls to existing, request-response oriented servers. However, such approach exhibits sensitivity to network latency since the calls are performed in a ping-pong fashion, as in synchronous RMI. 2 The alternative solution is to enforce ordering at the server side, which leads to improved communication throughput by enabling clients to streamline invocations. However, the burden is placed on the server-side middleware which may need to perform message reordering, track message identifiers, etc. The choice of an appropriate strategy is left to protocol providers. Due to the performance characteristics, server-side ordering is recommended unless prohibited by the protocol restrictions or the interoperability requirements.
3.4. Exception Handling The synchronous nature of RMI forces the caller of a remote method to handle possible failures before proceeding with further activities. This is no longer the case for asynchronous RMI, where failure notification – a special case of a completion notification – may be received by the application at an unspecified point in the future. In fact, that notification may be received after subsequent asynchronous calls were initiated on the same stub, often with an implicit assumption that those preceding calls were successful. Considering the example in Figure 4 again, and supposing that the “started” notification was not delivered due to a transient communication failure, the server would receive “completed” not preceded by “started” which might be unexpected by the application. The solution introduced in RMIX endorses the intuitive optimistic approach by making the asynchronous failure state sticky. That is, once a failure occurs within a given thread on a given stub, the communication provider must guarantee that all subsequent calls from that thread on that stub (including those already enqueued) are rejected until a clearance is given by the application, either by explicitly resetting the flag or by performing a synchronous call and dealing with the rejection exception in the caller thread. Stickiness may be explicitly disabled on a stub if so desired. 2
The scope of the problem is reduced, though, since calls invoked via distinct stubs are not ordered, and because the application threads themselves are not blocked on communication.
A separate issue relates to application debugging. In the course of an asynchronous method call, many activities such as parameter marshalling, result unmarshalling, or execution of completion callbacks, are executed in the background, that is, by threads different than the invoker thread. Exceptions thrown during such activities are thus difficult to trace back to the original call. The issue is addressed in RMIX by utility classes that allow the establishment of the appropriate context association, so that the relevant exception stack traces are combined with the invoker thread’s stack trace. As with many other non-essential features which may have performance impact, this one may be disabled if desired.
3.5. Cancellation External conditions may sometimes motivate cancellation of a remote call in progress. Cancellation applies to synchronous calls as well as to asynchronous calls, though it usually takes a more natural form for asynchronous calls as it can be initiated from the invoker thread itself. Since the future result may be simply disregarded, cancellation in asynchronous RMI is non-essential. However, it may be very helpful in limiting unnecessary resource usage, e.g. if the remote method involves excessive computations or data transmission. A common reason for cancellation is the lack of responsiveness from the server, which is often caused by network failures. Hence, it is important that the cancellation does not try to contact the server (or perform other blocking operations) before returning control; otherwise, it could itself block indefinitely and require intervention.
Client
Server Disregard At Client-Side
Call Initiation Parameter Marshalling
Result Unmarshalling
Parameter Unmarshalling
Interrupt Client I/O Disregard At Server-Side
Method Call
Interrupt Server Thread
Result Marshalling
Interrupt Server I/O Ignore Result
Result Delivery
Figure 5: Cancellation at different stages of RMI call
Due to its inherently asynchronous nature, cancellation may require different actions at different stages of the call (Figure 5), or it may fail – for instance if the task has already completed, or if it cannot be cancelled for any other
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
interface Cancellable { /** @return false if the task could not * be cancelled, true otherwise */ boolean cancel(boolean mayInterruptIfRunning); boolean isCancelled(); boolean isDone(); }
Figure 6: “Cancellable” interface reason. It is commonly expected and intuitive that cancellation which fails does not have side effects. In terms of quality of service, two approaches are possible. In the conservative approach, cancellation is defined to be successful only if it can assert that the system is put into a state equivalent to one where the cancelled operation has never been initiated. In the best effort approach, such guarantees are relaxed. Unless immediately deemed impossible, the cancellation returns indicating success, allowing the system to continue cancellation efforts in the background, possibly in coordination with the server. There is no support for cancellation in standard Java RMI (as of Java 1.4). Interrupting the invoker thread may have a desired effect in some Java VMs, but the results are not consistent and relying on them is discouraged. In RMIX, cancellation is an optional feature of the communication provider. If supported, the future object returned by the provider from an asynchronous call should implement the Cancellable interface (Figure 6; again, based on java.util.concurrent [8]). That interface enables support for both conservative and best effort approaches. If the mayInterruptIfRunning parameter is false, cancellation may succeed only if the client-side middleware can guarantee that the call will never start. Often, it is attainable up to the moment the call arguments are fully marshalled. If the parameter is set to true, the system is allowed to perform more aggressive cancellation attempts, e.g. requesting the server to disregard the task, or even to interrupt an appropriate thread if the task is already executing. In the interest of flexibility, RMIX does not mandate that providers furnish elaborate cancellation policies, leaving it as a quality of implementation. In fact, the simplest acceptable (albeit not very useful) policy is to always fail.
3.6. Security Considerations In Java, security control is bound to threads of execution via the concept of protection domains and the “access controller” mechanism [12]. At any moment, access control privileges are determined by the content of a current invocation stack. As previously mentioned, asynchronous RMI delegates some application-level operations (namely parameter marshalling, result unmarshalling, and execution of completion callbacks) to background threads. Those background threads usually are system threads running with very
wide or even unlimited access privileges. This opens a potential security vulnerability: for instance, a malicious user could execute sensitive code within a system-level access control context by providing this code as a completion callback to an asynchronous call. To avert this vulnerability, RMIX providers must ensure appropriate propagation of access control context from the invoker to all background activities performed on behalf of that invoker. The invoker context must be stored at the time of invocation, and all the background activities must be performed within this stored context. RMIX provides utility classes which simplify implementation of this feature.
4. Implementation In order to simplify development of service providers, RMIX supplies a collection of abstract base classes offering generic implementations of many aspects of RMIX behavioral specifications. One such class, AbstractAsyncInvoker, provides complete coverage of a client-side asynchrony, requiring the provider merely to supply an implementation of a plain synchronous call and the means to perform asynchronous parameter marshalling. The base class handles aspects of background thread pooling, configurable data consistency and exception handling policies, call ordering, cancellation, and security context propagation, as described earlier. Since the approach is based on client-side call ordering, it does not require any changes to the wire protocol or to server-side middleware; thus, asynchronous calls may be enabled on top of existing protocols and servers. At the time of this writing, there exist three concrete provider implementations based on AbstractAsyncInvoker, using JRMP [14], SOAP [3], and RPC [5] as the underlying transport protocols. As has been discussed earlier, performance considerations suggest that server-side ordering should be preferred to client-side ordering if allowed by the protocol. Implementing server-side ordering requires more substantial effort which cannot be easily generalized into a base class (due to the very fact that it has to be addressed at the protocol level). Currently, RMIX includes one provider implementation supporting server-side ordering on top of a slightly modified version of the RPC protocol. Thorough performance evaluation of the implementations outlined above is beyond the scope of this paper; however, it is the authors’ observation that the performance patterns are similar to those described elsewhere [10, 15], confirming the performance advantages of asynchronous RMI.
5. Related Work Performance issues of synchronous RMI are well known, and solutions have been proposed in the literature [10,
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
6, 15]. ARMI [10] is one of the first such approaches; like RMIX, its assumes server-side application transparency and uses asynchronous interfaces at the client side, following specific naming conventions. Completion notification is achieved using mailboxes which combine the functionality of futures, callbacks, and result queues. An alternative approach [6] does require modifications at the application server-side, and it is based solely on futures. Yet another solution [15] is limited to one-way calls, thus it does not provide any completion notification mechanism. Among those approaches, interoperability with the JRMP protocol is usually retained, and specialized tools are provided to generate modified stub classes. In some cases, exhaustive performance results have been presented. However, the issues of call ordering, exception handling, and security, were not addressed; some other issues (e.g. related to data consistency) were sometimes implied but not explicitly discussed. None of these solutions utilized dynamic stubs or supported multiple invocation protocols. ProActive [2] is a Java-based framework for parallel and distributed computing. It features asynchronous calls with precise semantics, completion notification based on futures, and well-defined exception handling model. However, the threading model of ProActive (based on the active object design pattern) is distinct from the one embraced by RMI. Moreover, unlike RMIX which is a general-purpose communication library, ProActive is a complete distributed computing framework and it imposes certain programming model. Aspects of concurrency have been recently addressed by CORBA [1]. The newly introduced Asynchronous Method Invocation (AMI) mechanism features server-side transparency, one-way calls with various synchronization modes, and asynchronous calls with completion notification via futures and callbacks. However, issues of call ordering, cancellation, or security are not addressed by CORBA at the time of this writing.
6. Conclusions and Future Work In this paper, semantic issues related to the asynchronous RMI paradigm are discussed. These issues include invocation syntax, data consistency policies, call ordering guarantees, exception handling, cancellation, and access control. Solutions are proposed, and their realization within the RMIX communication framework for Java is described. In the 1.5 “Tiger” release, the Java language will be enhanced with a metadata facility that may provide better syntactical ways to devise asynchronous interfaces than naming conventions used currently. Also, interruptible I/O introduced in Java 1.4 has created new possibilities for asynchronous cancellation which are not yet utilized in the current RMIX provider implementations. These and other possible enhancements will be investigated in future work.
References [1] D. Brunsch, C. O’Ryan, and D. C. Schmidt. Designing an efficient and scalable server-side asynchrony model for CORBA. In ACM SIGPLAN workshop on languages, compilers and tools for embedded systems, pages 223–229, Snow Bird, Utah, US, 2001. Available at http://portal. acm.org/citation.cfm?id=384227. [2] D. Caromel, W. Klauser, and J. Vayssiere. Towards seamless computing and metacomputing in Java. In G. C. Fox, editor, Concurrency Practice and Experience, volume 10, pages 1043–1061. Wiley & Sons, Ltd., Sept-Nov 1998. Available at http://www-sop.inria.fr/oasis/proactive/. [3] D. Box et al. Simple Object Access Protocol (SOAP) 1.1. http://www.w3.org/TR/SOAP/. [4] I. Foster, C. Kesselman, J. Nick, and S. Tuecke. The physiology of the grid: An open grid services architecture for distributed systems integration, Jan. 2002. Available at http://www.globus.org/research/papers/ ogsa.pdf. [5] IETF. ONC Remote Procedure Call (oncrpc). http://www.ietf.org/html.charters/ oncrpc-charter.html. [6] K. E. Kerry Falkner, P. D. Coddington, and M. J. Oudshoorn. Implementing asynchronous remote method invocation in Java. In Parallel and Real Time Systems (PART’99), Melbourne, Nov. 1999. Available at http://www.dhpc. adelaide.edu.au/reports/072/abs-072.html. [7] D. Kurzyniec, T. Wrzosek, V. Sunderam, and A. Slominski. RMIX: Multiprotocol RMI framework for Java. In Java Parallel Distributed Computing Workshop, Nice, France, 2003. Available at http://www.mathcs.emory.edu/dcl/ rmix/rmix.pdf. [8] D. Lea. Concurrency JSR-166 interest site. http://gee. cs.oswego.edu/dl/concurrency-interest/. [9] Object Management Group. http://www.omg.org. [10] R. Raje, J. Williams, and M. Boyles. An asynchronous remote method invocation (ARMI) mechanism for Java. Concurrency: Practice and Experience, 9(11):1207–1211, Nov. 1997. Available at http://www.cs.iupui.edu/ ˜rraje/pub/rmi.ps. [11] RMIX Home Page. http://www.mathcs.emory. edu/dcl/rmix/. [12] Java security home page. http://java.sun.com/ j2se/1.4/docs/guide/security/. [13] K. Seymour, H. Nakada, S. Matsuoka, J. Dongarra, C. Lee, and H. Casanova. Overview of GridRPC: A remote procedure call API for grid computing. In Grid Computing - GRID 2002, Third International Workshop, pages 274–278, Baltimore, MD, USA, Nov. 2002. Available at http://citeseer.nj.nec.com/ seymour02gridrpc.html. [14] SUN Microsystems. Java Remote Method Invocation: Distributed computing for Java. http://java.sun.com/ marketing/collateral/javarmi.html. [15] T. Sysala and J. Janecek. Optimizing remote method invocation in Java. In 13th International Workshop on Database and Expert Systems Applications (DEXA’02), Aix-en-Provence, France, 2002.
Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)
0-7695-2132-0/04/$17.00 (C) 2004 IEEE
