architecture is its ability to select the elements of a protocol stack dynamically at .... most interactions are bi-directional, endpoints require a transmission interface ...
A Configurable Protocol Architecture for CORBA Environments
Stephen Crane and Naranker Dulay Department of Computing Imperial College of Science, Technology and Medicine 180 Queen’s Gate, London SW7 2BZ, UK {jsc, nd}@doc.ic.ac.uk
Abstract This paper describes a flexible architecture for building the protocols required to allow interaction between distributed objects in a CORBA environment. A key feature of the architecture is its ability to select the elements of a protocol stack dynamically at bind-time depending on the properties of the interface being accessed. This permits multiple object-invocation protocols to coexist such that a system may support local, intra-ORB and inter-ORB protocols and allows the selection of the most appropriate protocol at run-time. In addition, the architecture is capable of supporting “non-standard” interaction protocols such as multimedia streams. The paper outlines the architectural principles used and describes an efficient implementation of the CORBA Internet Inter-Orb Protocol. Keywords: Distributed Interoperability, CORBA.
Objects,
Protocol
Stacks,
1. Introduction Modern distributed software development environments are exhibiting an increasing trend towards standardisation and interworking. Notable in this field are the efforts of ISO in its Reference Model for Open Distributed Processing, RM-ODP [1] and the Object Management Group’s Common Object Request Broker Architecture, CORBA [2]. Independent of this standardisation work are the demands on distributed systems to carry an increasing variety of data, for example, to carry data with timedependent characteristics such as multimedia streams and to inter-work with legacy systems such as SQL databases and TP monitors. This has necessitated the development of a plenitude of ad hoc transport sub-systems, each dedicated to the maintenance of a particular quality of service, in response to an application’s temporal and legacy requirements. It is no longer possible for distributed system
designers to supply a single transport which will meet all needs, the semantics of the data are too diverse. Fundamental to the operation of a distributed system is the establishment of communication paths between its components, or binding. Essentially this involves the initialisation of a sender-side endpoint with the distributed address of its receiver-side peer. Since the format of the distributed address is tightly coupled both to the native format of the distributed system and to the transport system which recognises it, we believe that a rigorous approach to the act of binding establishment will lead to a flexible communications architecture in which both inter-working and the support of multiple transport protocols are elegantly supported. The structure of the remainder of this paper is as follows. In section 2, we outline the principles which influenced the development of the architecture and the structure of the resulting communications system. Section 3 describes the CORBA inter-ORB protocol and the structure of our implementation of it. It also describes the structure of Regis’ native intra-ORB protocol and demonstrates the comparative performance of Regis and a commercial ORB. Section 4 places this work in context with related work and section 5 concludes with future directions in which we intend to take this work.
2. Protocol Architecture This section describes the salient features of a communications architecture which is capable of supporting multiple styles of binding between different types of interaction classes over multiple transport protocols. The different styles of binding are first defined; then the act of creating a binding is described; finally we present the elements of the communications architecture explicitly designed to support the model of binding.
2.1 Binding We define binding as the establishment of a communication between components. The binding connects the endpoint where a service is required to that where it is provided.
Client required
Server provided
For communication to take place, the endpoints being bound must support the same interaction style and take on complementary roles. For example, a function and a function pointer have complementary roles but a function pointer may only be bound to a function with the same signature. Bindings are potentially many-to-one, one requirement may only address a single provision, but many requirements may address the same provision, thus preserving the semantics of procedure and variable name binding. Binding occurs dynamically between endpoints with no prior knowledge of each other. It is a basic requirement of re-useable components that they have as little direct knowledge of their environment as possible. In order to support dynamic binding, there must be some mechanisms of communicating the location of a provided endpoint to where it is required. In our architecture three such mechanisms are supported: first-party binding, third-party binding and back-binding. We use the term interface reference to denote an object which is transmissible between address spaces and contains all of the information required to establish a binding to its referend. 2.1.1 First-party binding First-party binding1 occurs when a client component obtains an interface reference to a provided service and uses it to bind one of its required interfaces. While bindings may be established in other ways [4], in this paper we will concentrate on first-party binding as it is the only style required by the CORBA specification. 1
2
References which support first-party binding are typically typed by the signature of their referend in order to enforce type compatibility. The origin of the interface reference is not defined but the client commonly obtains it from an interface trader or as a result of a previous interaction.
2.1.2 Third-party binding Third-party binding is performed by an entity (the third-party) which is neither the client nor the server. The third-party obtains a reference to the server’s interface, binds to the client’s binding service, and requests the latter to bind the specific required interface using the reference. Third-party binding is therefore at a conceptually higher level than first-party, since it requires first-party binding to perform its function. Examples of third-party binding are less common than first-party. Environments such as Regis [3] which require programmers to provide an explicit description of program structure use third-party binding to establish an initial component configuration. The idiom is also found in systems which allow controlled modification of their structure by external managers [4]. References involved in third-party binding are not typed by their referends; type checking is typically performed at a higher level, by the configuration language compiler or the management interpreter. 3
2
1
2.1.3 Back-binding The great majority of interaction in a distributed system is two-way. A client issues a request and at some later time, a server returns one or more replies. Since our model of binding is many-to-one, and many requests can be outstanding at a server’s interface, each request must store a return path or back-binding to the client. Conceptually this is created from a reference to a private service contained in the client’s endpoint.
1
2
2.1.4 Interface References Previous sections have described the motivations for dynamic binding and situations where they occur. This section describes how bindings are created from references. References in general have opaque structure: it is not possible to infer anything about the information or names contained in a reference besides the fact that there must be one or more contexts in which some or all of the information makes sense. The act of binding then consists 1
The first-party in a binding relationship is the client, the second-party the server, while a third-party is an external agent which knows, and is trusted by, the first and second parties.
of identifying these contexts and requesting them to create the elements of the binding. We name these binding elements protocols. A strongly-typed reference is a list of names each of which, by its type, identifies a unique protocol layer in the desired communications system. Construction of a compatible binding is performed by a factory which supports the complete type of the reference. Strongly-typed references thus guarantee that a compatible communications sub-system will be created under the client’s endpoint. However, since the structure of strongly-typed references is completely known at compile-time, the re-usability of components in which they occur is lessened.
2.2 Configurable Protocol Stacks In our architecture, a binding is supported by a linear stack of protocols under each of the bound endpoints. Context independence, and hence re-use, is obtained by requiring each protocol to conform to an abstract interface and encouraging ‘derived associations’ between protocols. At the top and bottom of the stack are the endpoints and devices respectively. Endpoints provide synchronisation with user-level threads while devices manage the interface to the underlying operating system or hardware device itself. 2.2.1 Linear Protocol Stack The figure shows a bi-directional protocol stack. The interface provided by a particular endpoint defines the style, or component programmer’s view, of the interaction. Since most interactions are bi-directional, endpoints require a transmission interface (Session::Client) and provide an
Endpoint Interaction interface Session::Client
Upcall
Protocol Session
Upcall::Client
Session::Client
Upcall
Device Session
Upcall::Client
Device management
interface (Upcall) at which incoming data is delivered. Protocols are intermediate objects in the stack. On their top side, they provide a transmission interface to their neighbour and require a reception interface, while on their bottom side, they require a transmission interface and provide a reception interface. Device objects must possess a top-side interface which conforms to that expected by protocols’ bottom-sides. Internally, of course, they translate invocations on this interface to device-specific operations. In our model, protocol stacks are configurable: in addition to the end-to-end data path, they provide the ability to issue configuration commands to one or more protocols in the stack, query them for aspects of their state, or receive notification of exceptional conditions. 2.2.2 Protocol implementation The class structure of the principal elements of a communications system which implements this model is shown in the OMT diagram [5] below. The core abstractions are Session, Upcall and their accessors Session::Client and Upcall::Client. An accessor’s role is to manage one or more references to services provided by the neighbouring protocol layer. It provides one operation, bind, permitting initialisation of its reference(s). The Session class defines the interface for downward movement of data in the protocol stack. End-toend data is transmitted using the transmit operation, while the protocol layer’s state is modified and interrogated by a higher layer via configure and query respectively. The Upcall class defines the interface for upward movement of data. End-to-end data arrives via the deliver operation, while warnings of exceptional conditions arrive via notify. By combining pairs of these core abstractions, we can specify derived abstractions. A Device is a combination of a Session and an Upcall accessor. A transmit-only protocol, or DProto, combines a Session and its accessor, while a receive-only protocol, or UProto, is composed of an Upcall and its accessor. A normal bidirectional protocol, Proto, unsurprisingly combines a UProto and a DProto. Finally an Endpoint comprises a Session accessor and an Upcall.
Session::Client
Session
bind (Session *)
Upcall::Client
transmit (DBuf &) configure (Attribs &) query (Attribs &)
DProto
Upcall
bind (Upcall *)
Device
deliver (UBuf &) notify (Attribs &)
UProto
Proto
Endpoint
2.2.3 Dynamic Protocol Configuration Two main principles are reflected implementation of protocols stacks:
in
our
• that individual protocols should be lightweight, they should not adversely affect overall communications performance. • that stacks should be dynamic, they should allow protocols to be introduced or removed at any time during the lifetime of the connection. The dynamic aspect means that the state of the connection is represented by the presence or absence of certain protocols rather than state variables internal to the protocol elements themselves. This has the effect of simplifying other protocol layers in the stacks and making the state of the connection externally visible without breaking the encapsulation provided by each layer.
The combination of the two principles also leads to a greater degree of code re-use since many interactions share the same state transitions; for example all interactions which support third-party binding by a configuration language have the required property that access to them will block the accessor until binding has taken place. This property is provided by a re-useable bind protocol element which exists until binding has taken place and is thereafter destroyed, [6]; further, the bind protocol may make configuration decisions based on the locality of the interface endpoints being bound. 2.2.4 Multiplexors (Protocol Trees) Although the linear stack is an important buildingblock of the communications system, it cannot alone
describe all possible communication patterns. In our model of communication, the most general pattern is the tree. Branching of the tree occurs when a ‘naming boundary’ is crossed at a multiplexor. Examples of multiplexing are extremely common: UDP and TCP are multiplexed over IP; the TCP layer itself maintains a set of connections each identified by a port minor number. While individual protocol layers may or may not add headers to messages, multiplexors must always do so.
3. CORBA Inter-Operability Protocol The Object Management Group has recently formalised a set of protocols to allow implementations of its object request broker architecture to inter-operate, the socalled Inter-ORB Protocols [2]. In this section we describe an implementation of the CORBA Inter-ORB protocols using the architectural principles described in section 2.
3.1 CORBA Inter-ORB Protocols Two protocols comprise the OMG inter-operability specification: • the General Inter-ORB Protocol (GIOP) • the Internet Inter-ORB Protocol (IIOP) The GIOP defines: the format of IDL data on the wire, a small number of message types (and their associated headers) and makes general assumptions about the QoS characteristics of the transport layer underpinning it.
The IIOP defines the underlying transport to be TCP/IP and defines the structure of an Inter-operable
Object Reference, IOR, to consist of, among other things, a TCP/IP endpoint address and an opaque object key to identify a target object within the address space identified by the TCP/IP endpoint.
3.2 GIOP and IIOP Implementation The diagram below shows the structure of the protocol layers comprising our implementation of the IIOP. At lower layers of the protocol stack, all I/O is non-blocking and reactive. Threading and synchronisation only become concerns at the level of Proxy and Endpoint. Our programming platform, Regis, provides non-pre-emptive user-space threads but also utilises kernel threads where provided by the operating system.
ORB::Proxy
IIOP::Client
GIOP
ORB::Endpoint
IIOP::Server
GIOP
the send method of the ORB Proxy class, whose main function is to store the identity of its peer. Instances of the IIOP Client class are allocated perconnection, multiplexing requests from many proxies to the same server site. Its send method creates a ‘RequestHeader’ from the opaque target object identifier, the operation name and the marshalled request data. It returns a request identifier in order later to distinguish the reply. The bottom-side interface of Client conforms to the protocol stack combination of Session::Client and Upcall. The buffer containing the marshalled data and the RequestHeader then arrives at the Session interface of the GIOP layer. This layer is almost identical at either end of the connection but parametrised by its primary role. It builds a MessageHeader, stating the message’s type and size, and transmits it via the TCP layer. On arrival of the data at the server’s site, the TCP layer associated with this connection is upcalled. It reads all available data from the stream and passes it up to the GIOP layer. This is not a pure Proto layer because its Upcall interface is implemented by the Packetise class which breaks a stream of data into buffers, one per invocation. This is necessary because data belonging to an invocation might arrive in fragments, depending on the buffering provided by the kernels at either end of the connection. DProto
TCP::Device
Upcall
Upcall::Client
TCP::Device Packetise
3.2.1 Layer descriptions We describe the function of each layer in the IIOP protocol stack by tracing the path of an invocation between
Upcall
Session::Client
GIOP
When all of the data has been assembled, the GIOP layer passes it up to the IIOP Server layer which demultiplexes the invocation to its destination Endpoint using the RequestHeader. An Endpoint is parametrised by a Handler implementing one of several processing strategies:
IIOP::Client send (…) getReply (…)
IIOP::Proxy send (…) getReply (…) deliver (…)
a bound client and server. The client initiates a remote method invocation by calling the corresponding method of the IDL-generated proxy. This marshalls its arguments into a buffer and calls
• if the invocation is merely of the ‘get attribute’ type, it is performed eagerly and the reply transmitted. • if it requires more processing time, or if the component which owns the interface wishes explicitly to drive the invocation, it is deferred. • if greater concurrency is required, the invocation is passed to one of a pool of pre-allocated threads for processing. However it is processed, the invocation eventually completes and a reply returned to the invoker. The Endpoint passes it to the Server layer, to whose
Session interface it has retained a reference providing a ‘back-binding’. The Server adds a ‘ReplyHeader’ to the data buffer, identifying the request and return status and passes the buffer to the GIOP layer. This adds the MessageHeader and forwards it to the TCP layer for transmission. Upcall
creates a new TCP Device and passes it the invocation data and a pointer to its upcall which is in fact the IIOP Factory. When it receives the data buffer, the IIOP factory creates new Server and GIOP layers and binds them to the TCP Device, allowing the next invocation to proceed independently of it.
Regis::Referend
3.3 Regis Intra-ORB Protocol IIOP::Server
We demonstrate the flexibility of our architecture by describing our intra-ORB protocol. This is based on UDP/IP and already supports a wide variety of interaction styles, for example message ports, Ada-style entries and event disseminators. The connectionless nature of UDP/IP results in a slightly different structure at lower levels of the protocol hierarchy. A second level of multiplexing is required now to replace the connection-oriented semantics of TCP/IP. This is supplied by the Regis multiplexor bound to the UDP device.
IIOP::Endpoint
Handler
Eager
Deferred
...
At the client side, the reply filters up through the TCP and GIOP layers to the Client. From the request identifier stored in the ReplyHeader, it determines which Proxy transmitted the original request and delivers the reply data and return status to it. The Proxy then makes any thread blocked on the reply runnable. When it runs, the thread directs its Proxy to unmarshall the reply data (or throw an exception) and the invocation is complete. 3.2.2 Binding and Stack creation Creation of IIOP protocol stacks is managed by an IIOP object which is present in every address space. A server creates a reference (an IOR) to an interface from the interface’s endpoint and its type. The TCP endpoint contained in the IOR is that of the IIOP instance at the server’s site. A client obtains the IOR and uses it to bind a proxy by asking its local IIOP Factory to construct (or re-use) a client-side protocol stack consisting of Client, GIOP and TCP layers. It returns a pointer to the Client element at the top of the stack. When the first invocation between address spaces reaches the server’s site, it arrives at the ‘listening’ TCP endpoint in the server’s IIOP instance. This TCP sub-layer
IIOP::Client GIOP IIOP::Server
The intra-ORB client layer adds an interaction-specific protocol header to outgoing data, naming the destination endpoint, and providing a return path for replies. The Regis multiplexor adds a Regis protocol header comprising a byte-order tag and the name of the remote service. ORB::Proxy
ORB::Endpoint
IntraORB::Client
IntraORB::Server
Regis::Mux
Regis::Mux
UDP::Device
UDP::Device
When data arrives at the server’s site, the UDP device builds a back-binding containing the client’s UDP endpoint, and upcalls the demultiplexor with the data. The
Upcall
Upcall::Client
Device
IIOP::Factory
TCP
TCP::Device
#Create (…) #Release (…) deliver (UBuf &) IIOP
demultiplexor reads the Regis protocol header and uses it to forward the message to the intra-ORB service layer which in turn reads the intra-ORB protocol header and from it, demultiplexes the request to the destination endpoint. This protocol further differs from the IIOP by using an integer to identify the method to invoke. Two advantages accrue from this: first, the invocation overhead is independent of the number of operations in the interface; and second, the size of the protocol header is constant, allowing efficient marshalling.
While the use of UDP/IP places an upper bound on the size of the data which may be sent between endpoints, it has proved sufficient for the majority of our applications. Furthermore, since it has a well-defined protocol interface, its behaviour may easily be modified by the addition of fragmentation and reliable message delivery layers if required.
3.4 Performance Preliminary measurements of the performance of our implementation are very encouraging. To test the efficiency of our protocol layers, a client performed several thousand ‘null’ invocations on a server’s interface, passing it a sequence of octets which varied in length. 3.4.1 Inter-ORB protocol The table shows the minimum time recorded for the different data sizes passed using the IIOP, and for comparison the time taken by a commercial ORB, Orbix2.0.1, [7], running a similar program with an identical IDL interface. The platform was a lightly-loaded Sun SS-20 Sequence length Regis Orbix 1 1.698 3.014 10 1.681 3.037 100 1.714 3.007 1000 1.948 3.103 10000 4.069 4.326 running Solaris 2.4. The Regis program was executed under SunOS binary emulation while the Orbix program was explicitly built for Solaris. 3.4.2 Intra-ORB protocol The same tests as above were repeated for a program which used a proprietary intra-ORB protocol. The Regis Sequence length 1 10 100 1000
Regis 1.275 1.288 1.331 1.550
Orbix 1.843 1.880 1.852 1.959
ORB employed the lightweight protocol described in the last section, while we believe Orbix continued to use TCP. The results are shown in the accompanying table. The sequence length was bounded in the Regis case by the maximum length of a UDP packet. 3.4.3 Discussion One of the features of our implementation which enhances performance is the buffer class. It relies on kernel support for ‘scatter-gather’ I/O to minimise data copying when a header is to be added to a message. In the common case, this allows an invocation to be assembled from an ordered list of pointers to automatic data. Another contributory feature is the reactive input behaviour which requires no thread switch until the data reaches the endpoint itself and, in this example, none then. These tests were run on the same machine for several reasons. In the inter-ORB case, nothing would have been gained by distributing the example, it is assumed that the kernel responded to each application identically, so that what is being compared is the relative performances of the two run-time systems. In the intra-ORB case, nondistribution has the effect of ironing out performance differences between UDP/IP and TCP/IP, resulting again in a direct comparison between the two run-times. In passing, it is worth noting the difference between the performance of Orbix’s native protocol and its implementation of the IIOP, although they are both out-performed by our IIOP implementation.
4. Related Work The concept of layered protocols possessing abstract interfaces originates with Ritchie [10] and has also been exploited by the X-kernel [8] and Horus [9] which feature light-weight and reusable communication protocols. Our architecture extends these systems by offering support for configurable protocol stacks and trees and by supporting multiple distributed object-invocation protocols such as those found in CORBA and in Regis. Our architecture’s reactive I/O model also employs design patterns [11] for abstracting the key functionality of configurable protocol stacks. Inter-stack operations, such as those supported by the ‘configure’, ‘query’ and ‘notify’ operations, are examples of the chain-of-responsibility pattern while the abstract interfaces defined by ‘session’ and ‘upcall’ are facades. Parameterisation of a remotelyinvokable object by an ‘invocation handler’ is an example of the strategy pattern. The work is described in detail in [14]. Our model, although independently developed, is functionally equivalent to Schmidt’s Reactor pattern [12] while our TCP layer is similar to his Connector-Acceptor [13].
5. Conclusion In this paper we have presented an architecture for building a diverse collection of configurable objectinvocation protocols for use in a CORBA environment. The architecture can also support “non-standard” protocols such as those needed for time-dependent multimedia streams and legacy access to TP monitors and databases. The abstraction layers of the architecture are design-pattern based and encourage light-weight, layered and re-usable protocol code. The architecture needs to be enhanced in some areas. The most pressing requirement is to employ the ‘shortcircuit’ binding pattern [14], to support efficient communication between co-located interfaces. Another desirable feature is support for third-party binding and the configuration language Darwin [3]. In a more general scheme of things, distributed bindings between objects, and our view of them, do not differ substantively from bindings between other interactions such as ports and event disseminators. In essence, we are employing the IDL compiler as a factory of configurable RPC-like interaction classes. Our prototype implementation (in common with Orbix) does not easily allow selection of the desired protocol configuration since references to interfaces are statically typed by the desired transport, with a corresponding effect on component re-usability. Run-time selection of transport protocols will remedy this inflexibility in the future. In conclusion, we have presented a flexible protocol architecture for CORBA environments. The architecture’s flexibility has been demonstrated by its application to the CORBA IIOP. As demonstrated by the prototype’s performance figures, this flexibility does not come at a cost.
2.
3.
4.
5. 6.
7. 8. 9.
10. 11. 12.
6. Acknowledgements The authors acknowledge the financial support of British Telecommunications plc. We also acknowledge the contribution of Halldor Fosså of the back-end to the IDL compiler and many stimulating discussions with our colleagues in the Distributed Software Engineering section, in particular Jeff Kramer, Jeff Magee, Nat Pryce and Morris Sloman.
7. References 1.
ODP Reference Model Part 1, Overview. Secretariat: ISO/IEC JTC1/SC21/WG7. Standards Association of Australia, PO Box 1055, Strathfield, NSW, Australia 2135, May 1995.
13.
14.
The Object Management Group. The Common Object Request Broker: Architecture and Specification, Revision 2.0. OMG Headquarters, 492 Old Connecticut Path, Framington, MA 01701, July 1995. J. Magee, N. Dulay and J. Kramer. A Constructive Development Environment for Parallel and Distributed Programs. In Distributed Systems Engineering, 1(5): 304312, September 1994. IEE/IOP/BCS. S. Crane, N. Dulay, H. Fosså, J. Kramer, M. Sloman. Configuration Management for Distributed Software Services. In Integrated Network Management IV, Proceedings of ISINM-95, Santa Barbara, USA. Editors A. Sethi, Y. Ranoud, F. Faure-Vincent. May 1995. Chapman and Hall. J. Rumbaugh, et al. Object-Oriented Modelling and Design. Prentice-Hall International Inc. New Jersey, 1991. ISBN 013-630054-5. S. Crane, J. Magee and N. Pryce. Design Patterns for Binding in Distributed Systems. Presented at the OOPSLA-95 Workshop on Design Patterns for Concurrent, Parallel and distributed Object-Oriented Systems. October 1995. Austin, Texas, USA. IONA Technologies Ltd. The Orbix Architecture. January 1995. http://www.iona.ie S. W. O’Malley and L. L. Peterson. A Dynamic Network Architecture. ACM Transactions on Computer Systems, 10(2): 110-143, May 1992. R. van Renesse, T. M. Hickey and K. P. Birman. Design and Performance of Horus: a lightweight Group Communication System. Technical Report, Department of Computer Science, Cornell University, Ithaca, New York, 1994. D. M. Ritchie. A Stream Input-Output System. AT&T Bell Laboratories Technical Journal, 63(8): 1897-1910, October 1984. E. Gamma, R. Helm, R. Johnson and J. Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1994. ISBN 0-201-63361-2. D. C. Schmidt and T. Suda. The Service Configurator Framework: An Extensible Architecture for Dynamically Configuring Concurrent, Multi-Service Network Daemons. In Proceedings of the Second International Conference on Configurable Distributed Systems, March 21-23 1994, Pittsburgh Pennsylvania, USA. IEEE Computer Society. D. C. Schmidt. Acceptor and Connector: Object Creational Design Patterns for Actively and Passively Initialising Network Services. Presented at The European Pattern Language of Programs Conference, July 10-14, 1996, Kloster Irsee, Germany. N. Pryce and S. Crane. A Uniform Approach to Configuration and Communication in Distributed Systems. In Proceedings of the Third International Conference on Configurable Distributed Systems, May 6-8 1996, Annapolis Maryland, USA. IEEE Computer Society.
