Design Patterns for Binding in Distributed Systems ... - CiteSeerX

Design Patterns for Binding in Distributed Systems Stephen Crane

Je Magee Nat Pryce jnm, [email protected] Department of Computing, Imperial College of Science, Technology and Medicine, London SW7 2BZ, UK. September 1, 1995 fjsc,

Abstract

1 Introduction Regis [1] constructs a distributed application from a set of components. Each component provides services which can be accessed by other components and may require access to services from other components. Components may be composed hierarchically to form composite components. Each component has a clearly de ned interface in terms of the services it provides and those it requires. Components execute concurrently. Figure 1 depicts a simple system consisting of three components.

The paper reports on the design patterns we have found useful in Regis, a toolkit for constructing distributed applications. Regis supports the construction of a system from a set of components. Components interact by accessing local and remote services. Access is implemented by one of a set of communication protocols. The protocol can be chosen and subsequently changed at runtime. Overall application construction is driven by a con guration description which speci es the application structure as a graph of interconnected components. This structure can also change at runtime. The design patterns we describe here have been used to satisfy the requirement for a exible support environment which can accommodate multiple communication protocols and which satis es the requirement for dynamic inter-component binding. The protocol structuring pattern and the associated binding pattern seamlessly integrate both rstparty (client) binding and third-party (manager) binding to services and have general application in the con guration management of distributed systems.

source

filter out

in

sink out

in

Figure 1: A pipeline. In the diagram, a lled circle represents a service provided by a component and an empty circle represents a service required by a component. A binding, represented by an arc, associates a required service with a provided service. Regis uses a well-de ned language called Darwin [2] with both a graphical and textual representation for describing these structures. For example the binding between the source and lter components would be declared in 1

2 The protocol framework

Darwin as:

bind source.out --

Fundamental to the operation of Regis in a distributed manner are the classes remobj and remref. The remref class implements a safe, ecient distributed pointer to objects derived from remobj. Components' service provisions are derived from remobj while service requirements are derived from remref. Establishing a binding between a requirement and a provision involves initialising a requirement with a remref to the provision. For a requirement to make use of its bound service, it must be able to communicate with it. Previous versions of the Regis runtime supported a single xed communication protocol (UDP/IP) managed by a thread dedicated to receiving incoming messages on the UDP port. This structure has suced to use Regis as an instructional testbed for students. Recent requirements of several collaborative projects, however, led to the decision to support multiple transport layers (i.e., ATM and Multicast-IP) and process incoming messages more eciently to support multimedia communication. To satisfy these requirements, the base classes of service provisions and requirements (sap | service access provision | and sar | service access requirement) were extended with dynamic, con gurable protocol stacks. The protocol layers channel data between the user-level threads implementing Regis components and the network devices managed by the underlying operating system. As such, they act as bi-directional chains of responsibility.

filter.in;

In general, bindings are many-to-one: many requirements may be bound to one service provision. A binding in Darwin results in a communication path at runtime between the component requiring the service and that providing the service. Regis services are implemented in C++ as instances of either system-provided or user-de ned interaction classes. Examples of system-provided classes are Ports, Streams and Events. Primitive components are implemented as lightweight active objects (derived from a class process). These active objects implement the service accessed through an interaction object. Darwin enables the speci cation of complex structures which are elaborated when an application is started and which may evolve as execution progresses. The initial system structure may be modi ed by external con guration management requests which may cause new components to be created and bindings to be changed. Components may be colocated in the same heavyweight operating system process or may be on remote hosts. Component location is decided at runtime. An application in Regis may use more than one communication protocol for accessing dierent services e.g. ATM and TCP/IP. The application con guration exibility supported by Regis requires a similar exibility in the execution support environment. In the paper we outline the object patterns we have found useful in designing and implementing a runtime architecture which satis es our exibility requirements. The next section outlines the pattern used to structure protocol layers. The following sections describe how the various binding requirements are integrated with this framework.

2.1 Abstract Interfaces

In our design, an object implementing a protocol layer can oer two interfaces to other layers: an interface through which the layer below it passes information up to be consumed and an interface through which the layer above 2

it passes information down to be transmitted. These interfaces are encapsulated in two abstract classes: uproto, used for passing data up the stack, and dproto used for passing data down the stack. Two classes of data which were to be passed through each interface were identi ed: data which is being communicated (i.e., is being transmitted or received); and data pertinent to the operation of the protocol layers (indications of exceptional events and con guration requests). Data to be communicated is passed around in buffer objects which allow protocols to add headers onto a message eciently and can be shared among protocols without copying.1 Indications of exceptional events and parameters to con guration requests are passed as attrib objects which are lists of name-value pairs. Additional methods are required to connect protocols | dprotos can be connected to a uproto above them and vice versa.

data from below and above. The interesting cases are at the top and bottom of the protocol stacks.

2.2 External interfaces

The bottom-most protocol layers handle the reception and transmission of data via Unix device descriptors. This means that they not only provide an interface to protocol layers above them (they derive from dproto but not uproto) but also have to handle I/O events occurring on the descriptors of their network connection. The handling of I/O events is an example of the reactor pattern [3] and is encapsulated within the io class. Therefore, the lowest protocol layers are multiply derived from both dproto and io classes. io oninput onwrite onexcept

class dproto { virtual int deliver_down( buffer*, remref*=0 ) = 0; virtual int configure( attrib* ) = 0; virtual int dchain( uproto* ) = 0; };

lowest layer

dproto deliver_down configure query dchain

mux

uproto deliver_up exception uchain

proto name depth

Figure 2: Class hierarchy of the lower protocol layers

class uproto { virtual int deliver_up( buffer*, sap*=0 ) = 0; virtual int exception( attrib* ) = 0; virtual int uchain( dproto* ) = 0; };

Because remref objects are used to reference services, multiple layers of protocol demultiplexing are not necessary. A remref to the remote service is passed down to the bottom-most protocol layer when a message is transmitted and included in the header for message. When the message is received, Most protocol layers derive from both itthat contains a local pointer to its destination. uproto and dproto, since they have to receive Services can therefore have a linear stack of 1 Under normal circumstances. protocols which process the incoming message. 3

An incoming message is delivered to a service by passing it up to the bottom protocol of the service's stack. This functionality has been encapsulated in the mux class and so can be reused above many transport protocols. A similar protocol is the umcast protocol which multicasts an incoming buer to one or more services; this makes multicast communication more ecient by allowing multiple services to share the same network endpoint. The topmost layers of the communication software pass received messages to components' threads, which accept messages using `guarded receive'. The sap class is derived from remobj (to be remotely referenceable) and msync (which implements guarded reFigure 4: A snapshot of a typical protocol conceive). guration aPort

stk

aProtostk

aFragproto

top

prev

bottom

next

aSeqproto

prev next

aMux

lower

aUDPproto

prev next

aUDPsingleton upper

lower

dproto

uproto

deliver_down configure query dchain

remobj

msync

deliver_up exception uchain

terns for performing con guration and recon guration using dynamic protocol stacks, described in the familiar \gang-of-four" style [4].

arm disarm

3 Con guration

remref bind waitvalid binding

An Object Behavioural pattern.

3.1 Intent sar

proto name depth new_proto

Con gure a binding between endpoints once, making optimisation decisions then instead of on every transmission.

sap

transmit

receive exception bottom

stk

stk

3.2 Also Known As

protostk

Software Integrated Circuit. Programming-inthe-large.

insert remove top bottom

3.3 Motivation

Figure 3: Class hierarchy of the upper protocol The simplest connection between component layers instances in a Darwin con guration descripThe next two sections present design pat- tion is that performed by the bind statement: 4

bind source.out --

filter.in;

proto deliver_down()

Whether the endpoints are colocated or distributed is not determined until runtime. Furthermore, the model dictates that transmission on an unbound endpoint block the invoker until binding has taken place. This caused the frequent appearance of the following fragment of pseudo-code:

bindproto deliver_down()

transmit (data): wait_valid (); if (peer_remote ()) remote_transmit (data); else local_transmit (data);

import deliver_down()

Figure 5: Class diagram for the bind protocol.

All transmissions barring the rst one therefore required two redundant tests: for validity and locality. Another common form of binding occurs in open systems where there is some degree of autonomy between client and server:

import source.out --

localproto deliver_down()

waits for the binding to be established before checking whether the provided endpoint is local and destroying itself. 

/service;

localproto

This is an ecient shortcut created by if the provided endpoint is in the same address space as the required one.

bindproto

It should be possible to implement the second form of binding in a similar fashion to the rst.

3.4 Applicability



Use transient con gurational protocols to create an application's initial structure in order to decouple structural issues from normal execution.

3.5 Structure

The class structure of con gurational protocols for binding and importation is shown in gure 5.

import

Augments the behaviour of bindproto by querying the nameserver for the required service before transmission. (Consistent behaviour is therefore guaranteed in the pathological case where the program imports a service which it has earlier exported.)

3.7 Collaborations

Figure 6 depicts the collaboration between bindproto and localproto during initial 3.6 Participants binding establishment. Before the rst transmission, the requirement side's protocol stack  bindproto Attached to the required side of the bind- consists of a bindproto element. When the ing before rst transmission. This class rst transmission is attempted, bindproto 5

aBindproto

ing it. It is the responsibility of the bind protocol to wake the thread once binding has taken place.

aLocalproto

wait_valid()

deliver_down()

FIRST

darwin_bind

TRANSMISSION

deliver_down()

3.10 Sample Code

Space limitations preclude the inclusion of source code.

local_send() SUBSEQUENT

TRANSMISSIONS

3.11 Known Uses

Many distributed programming environments Figure 6: Interaction diagram for the bind pro- use con guration languages to clarify application structure. Conic [5] and Durra [6] are tocol. examples. causes the invoker to be blocked until the endpoint has been bound by the con guration language (darwin bind). Once a valid service has been obtained, bindproto determines whether it is in the local address space. If so, it creates a localproto to short-circuit the remote protocol stack which was either speci ed explicitly in the con guration language or implicitly as a system default.

3.12 Related Patterns

3.8 Consequences

An Object Behavioural pattern. While the last section dealt with establishment of initial bindings, this one focuses on dynamically destroying and re-establishing bindings.

Protocols are an example of the chain of responsibility pattern. Service requirements are closely related to the proxy pattern. The classes import and bindproto are examples of decorators while localproto is an adaptor.

4 Recon guration

The motivation is for ecient communication in normal execution. Using the transient bind protocol to establish the validity of the binding and create an ecient shortcut for the case where the endpoints are colocated allows us to express the result of the test for validity as 4.1 Intent the absence of a bindproto element and the Provide a service whereby bindings between result of the test for locality as the presence of modules can be created and subsequently a localproto element. modi ed in a safe fashion: the cooperation of the programmer of the component owning the binding is required. 3.9 Implementation

Implementation of bindproto makes one critical assumption about the semantics of re- 4.2 Also Known As quired interfaces: a transmission on an un- (Third party-) bind service. Con guration bound interface will block the thread attempt- management service. 6

4.3 Motivation

4.6 Participants

Long-running programs often require `steer- We discuss those classes speci c to the rebind ing' to enable them to cope with changing en- protocol, enclosed in the dotted box in gure 7. vironments. In the model, steering comprises a combination of the basic operations create-  rebind Speci es the initial state of the protocol. component, create-binding, destroy-binding When transmission is attempted on this and destroy-component. Preservation of conelement, it creates the bind service and sistency is often a more pressing concern the bound and unbound states, switches than immediacy and the destruction operato the unbound state and destroys itself. tions must take this into account. 

4.4 Applicability

Use the rebind protocol to provide recon guration hooks which allow a manager to modify the structure of a running application such that consistency between endpoints is preserved.

4.5 Structure

Figure 7 contains an OMT class diagram for the recon guration protocol layer. proto deliver_down() configure()



sap

bindproto deliver_down()

unbound deliver_down() request()

bound deliver_down() request()

bound

current previous

7

An abstract class containing the interface required by the concrete classes bound and unbound. Stores a reference to bindsrv. Classes deriving from it are required to provide a routine in which they process (or defer) bind requests.



bindsrv

Figure 7: Class diagram for the rebind protocol.

bindstate

unbound

request()

rebind

Advertises a bind service interface in the nameserver. Forwards binding requests received on this interface to the current state, having rst ltered out, and responded to, requests for the path to the provided endpoint in the namespace. It maintains the current and previous states and switches between them on demand. It handles con guration requests, storing the value of the `safe' status.

 bindstate

deliver_down()

bindsrv

Manage the unbound state. When in this state, it is always safe to perform binding operations, either by the con guration language (the initial binding) or through the advertised bind service. When binding has been performed, this class initiates a transition to the bound state.

Manage the bound state. A transition out of this state can only occur when an unbind operation is processed. The last rebind operation received when the binding

is unsafe to recon gure is kept pending. Recon guration is driven by reception of a request when the binding is safe or by marking the binding safe. No test for possible recon guration is done on transmission in order to make normal operation as ecient as possible.

bind service (or from the initial con guration description). The transmission is then completed and the protocol enters the bound state. 3. An unbind request is received and forwarded to the bound element. If it is safe to change the binding, this is done and the protocol enters the unbound state once more.

4.7 Collaborations

Figure 8 depicts the collaborations between elements of the unbind protocol in an object interaction diagram. aRebind

aBindsrv

anUnbound

export()

set(anUnbound) deliver_down()

proto::deliver_down()

request()

INITIAL

BINDING

receive(bind)

set(aBound)

request() issafe()

set(anUnbound)

UNBINDING

receive(unbind)



Use of rebind allows the programmer to advertise long-running programs' bindings as mutable in response to changes in their external environment.



While an earlier implementation of the third-party bind service required the component programmer to mark the service reference as safe to modify, the protocol nature of rebind further necessitates co-operation of the other elements in the protocol stack. This is a direct consequence of the potential for protocol stacks to encapsulate state about some facet of a connection (e.g., buers awaiting acknowledgement in a sequenced `datagram' protocol). This requires protocol layers which possess state to acknowledge a `shutdown' command (stateless layers can ignore this).

FIRST

TRANSMISSION

deliver_down()

aBound

4.8 Consequences

Figure 8: Collaborations in the rebind protocol. Three phases of collaboration are shown. 1. A transmission is performed on the aRebind element. aBindsrv, aBound and anUnbound are created. The bind service is exported to the nameserver and transmission continues through the anUnbound element. 2. A transmission is attempted in the unbound state. The invoker is blocked until a bind request is received from the

4.9 Implementation

The implementation of rebind as a protocol element leads to it being transparent to the interaction class it manages. An alternative implementation: merging rebind functionality with the endpoint requires re-implementation of each class of endpoint, clearly undesirable in a system such as Regis in which endpoints are otherwise unconstrained. 8

4.10 Sample Code

Horus [10] have also exploited this concept, indeed the bind protocol described in x3 is properly a virtual protocol in the terminology of X. Commercial systems too have found this concept useful, Orbix [11] refers to protocols as proxies and lters. Ultimately, of course, all of these systems are indebted to Ritchie's original work on streams for the Eighth Edition of Unix [12]. Where this work can claim originality is in the application of protocols to the con guration and managed evolution of dynamic distributed systems, with minimal impact on the eciency of normal execution. While the original intent was `merely' to open our system to a variety of transports (e.g. ATM, reliable groupcast [13]) while retaining its exibility of interaction classes, we have found that the architectural support necessary also allowed us to re-engineer our ideas on con guration and evolution in a satisfyingly consistent way. The authors acknowledge the contribution of their colleagues in the Distributed Software Engineering Section during the formulation of ideas behind Regis. Further information on Regis may be found at http://wwwdse.doc.ic.ac.uk/~regis/regis.html.

Space limitations preclude the inclusion of source code.

4.11 Known Uses

Third-party binding is widely acknowledged in the literature as a requirement for interactive recon guration, [7]. In [8], Friedberg observes: \ the implementation of ecient and transparent recon guration requires a mechanism for an arbitrary module to create or destroy a binding between two other modules without their active participation. We call this facility third-party connect. \Recon guration provides the ability to change partners bound across an interface. If a module is not aected by a change of partners at all, then recon guration is transparent. If a module has to participate in changing partners then recon guration is intrusive, but still provides some isolation." Online recon guration is essential in all longrunning applications which must provide continuous service. Perhaps the earliest recognition of this need was in telephony. :::

References [1] J. Magee, N. Dulay and J. Kramer. Regis: A constructive development environment for Distributed Programs. IEE/IOP/BCS Distributed Systems Engineering Journal, 1(5): 304{312, September 1994. [2] J. Magee, N. Dulay, S. Eisenbach and J. Kramer. Specifying Distributed Software Architectures. To appear in the Fifth European Software Engineering Conference, Barcelona, September 1995. [3] D. C. Schmidt and T. Suda. The Service Con gurator Framework, An Exten-

4.12 Related Patterns

The rebind protocol is an example of the state pattern.

5 Summary and Status The notion of a protocol in the sense described here is not original. In terms of design patterns, it is familiar as a chain of responsibility. Research systems such as the X-Kernel [9] and 9

sible Architecture for Dynamically Con guring Concurrent, Multi-Service Network Daemons. In Proceedings of the Sec-

rus: a Lightweight Group Communications System. Technical Report, Department of Computer Science, Cornell Uniond International Workshop on Con gversity, Ithaca, New York, 1994. urable Distributed Systems, pps 190{201, [11] Orbix | a technical overview. Iona TechIEEE, Pittsburgh, March 1994. nologies, The O'Reilly Institute, Westland Row, Dublin 2, Ireland, July 1993. [4] E. Gamma, R. Helm, R. Johnson and J. Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addi- [12] D. M. Ritchie. A Stream Input-Output System. AT&T Bell Laboratories Technison Wesley 1994. ISBN 0-201-63361-2. cal Journal, 63(8): 1897{1910, October 1984. [5] N. Dulay, J. Kramer, J. Magee, M. Sloman and K. Twidle. Distributed System [13] C. Karamanolis and J. Magee. Con gConstruction: Experience with the Conic urable Highly Available Distributed SerToolkit. In Experiences with Distributed vices. To appear in Proceedings of the Systems (J. Nehmer editor), pps 187{212. 14th IEEE Symposium on Reliable DisSpringer-Verlag 1987. tributed Systems, Bad Neuenahr, Germany, September 1995. [6] M. Barbacci, D. Doubleday, et al. Building Fault Tolerant Distributed Applications with Durra. In Proceedings of the International Workshop on Con gurable Distributed Systems, pps 128{139, IEE,

London, March 1992. [7] J. Kramer, J. Magee and A. Young. Towards unifying Fault and Change Management. In Proceedings of the IEEE International Workshop on Distributed Computing Systems in the '90s, pps 57{

63, Cairo, 1990. [8] S. Friedberg. Transparent Recon guration requires a Third-Party Connect. Technical report 220, Department of Computer Science, University of Rochester, New York, November 1987. [9] S. W. O'Malley and L. L. Peterson. A Dynamic Network Architecture. ACM Transactions on Computer Systems, 10(2): 110{143, May 1992.

[10] R. van Renesse, T. M. Hickey and K. P. Birman. Design and Performance of Ho10