To appear J. Parallel and Distributed Computing Practices, 2000, Tech Note DHPC-060.
Remote Data Access in Distributed Object-Oriented Middleware K. A. Hawick∗
H. A. James
J. A. Mathew
31 December 1998
Abstract Efficient, scalable remote access to data is a key aspect in wide area metacomputing environments. One of the limitations of current client-server computing models is their inability to create, retain and trade tokens which represent data or services on remote computers alongwith the metadata to adequately describe the data or services. Most current client-server software systems require the user to submit all the data inputs that are needed for a remote operation, and after the operation is complete, all the resultant output data is returned to the originating client. Pipelining remote processes requires data be retained at the remote site for achieving performance on high latency wide area networks. We introduce the DISCWorld Remote Access Mechanism (DRAM), an integral component of our DISCWorld metacomputing environment, which provides the user and system with a scalable abstraction over remote data and the operations that are possible on the data. We present a formal notation for DRAM’s and discuss the implementation and performance of DRAM’s when compared with traditional client-server systems. Keywords: distributed computing, middleware, metacomputing, Java, CORBA, DISCWorld
1
Introduction
The increasing need for data access across wide area computer clusters poses an interesting problem in robust middleware design. A critical aspect is that of reliable data exchange between cooperating applications and users in a wide area environment. High latencies and network failure probabilities and the need for transaction traffic to traverse administrative boundaries make wide area systems more difficult to cater for than local area clusters. Software systems for building middleware to glue together applications are emerging in the form of the Common Object Request Broker Architecture (CORBA) [1]. These, together with user interfaces written with the Java programming language and packages provide a good solution for local area systems. These packages do not of themselves however address the issues of robustness, latency tolerance and other performance considerations that are of especial importance for wide area systems. We discuss this further in section 7.1. ∗ Distributed
& High Performance Computing Group, Department of Computer Science, University of Adelaide, SA 5005, Australia. Tel +61 08 8303 4519, Fax +61 08 8303 4366, Email
[email protected]
1
Remote Data Access in Distributed OO Middleware
2
We are developing a more general architectural framework for a wide area metacomputing middleware, known as Distributed Information Systems Control World (DISCWorld) [10, 11]. This framework provides an experimental testbed for some of the current research ideas for achieving performance and interoperability across wide area systems. In this article we present an important development of DISCWorld that provides a remote data access mechanism between participating nodes, using object-oriented software technology. The DISCWorld architecture consists of a number of peer based computer hosts, that participate in DISCWorld by running either a DISCWorld server daemon program or a DISCWorld compliant client program. DISCWorld client programs can be constructed using Java wrappers to existing legacy programs, or can take the form of a special DISCWorld client environment which runs as a Java applet inside a World Wide Web (WWW) browser. This client environment can itself contain Java applet programs (Java Beans) [5] which can act as client programs communicating with the network of servers. The peer based nature of DISCWorld clients and servers means that these servers can be clients of one another for carrying out particular jobs, and are able to broker or trade services amongst one another. Jobs can be scheduled [12, 13] across the participating nodes. A DISCWorld server instance is potentially capable of providing any of the portable services any other DISCWorld server provides, but will typically be customised by the local administrator to specialise in a chosen subset, suited to local resources and needs. The nature of the DISCWorld architecture means that we use the terms client and server somewhat loosely, since these terms best refer to a temporary relationship between two running programs rather than a fixed relationship between two host platforms. DISCWorld is targeted at wide area systems and applications where it is worthwhile or necessary to run them over wide areas. These will typically be applications that require access to large specialist datasets stored by custodians at different geographically separated sites. An example application might be a land planning one [3], where a client application requires access to land titles information at one site, and digital terrain map data at another, and air reconnaissance or satellite imagery stored at another site. A human decision maker may be seated at a low performance compute platform running a WWW browser environment, but is able to pose queries and data processing transactions of a network of DISCWorld connected servers to extract decisions from these potentially very large datasets without having to download them to his own site. This is shown in figure 1. Our vision for DISCWorld is an integrated query-based environment where users may connect to a cloud of high performance computing (HPC) resources and request data retrieval and processing operations. Users themselves may only be connected into the cloud by a low bandwidth network link such as that provided by a modem line. This action-at-a-distance query-based approach appears to be appropriate for decision support applications where the user is provided with a collection of application components that run in situ in a WWW browser and help him control remote HPC resources. Much of our research to date has considered the multi-threaded software daemon (DWd) that runs on each participating DISCWorld service providing host [10]. In this paper we describe our work in devising an OO data transport and exchange mechanism to enable action-at-a-distance computing. The problem of managing the data exchange between servers and client programs is a non trivial one, but can be significantly eased through the use of object-oriented remote data pointers. In DISCWorld we refer to these pseudo-pointers as DRAM’s (DISCWorld Remote Access Mechanisms). DRAM’s are not the same as full data pointers in a programming system, but are constrained in certain ways to cope with the transactions typical for moving large data sets, and usually only moving the references to them, between applications running on widely separated hosts. The notion of DRAM’s is developed in section 2, with an outline of their various
Remote Data Access in Distributed OO Middleware
3
Runs DWd
Runs DWc Client
Logical Network
Application
Runs DWd
DISCWorld Architecture
Runs DWd
Cloud of Cooperating HPC Resources
Figure 1: DISCWorld Architecture. A number of cooperating server hosts communicate and share work, brokered by the DISCWorld daemon. Each is capable of acting as a gateway for client programs running on server hosts or as specialised graphical interface clients.
properties and uses. We describe a more formal notation for DRAM’s in section 3, our implementation of DRAM’s in our prototype DISCWorld framework in section 4 and our strategy for naming DRAM’s in section 5. We outline a performance model for the expected behaviour of DRAM’s and compare this with some experimentally measured performance behaviour in section 6. In section 7 we compare DRAM’s in the context of DISCWorld with other data exchange mechanisms in other research systems and outline what we believe are worthwhile future research directions.
2
DISCWorld Remote Access Mechanism
The DRAM idea is best explained through an example. We first describe the use of DRAM’s as pointers to data entities. Consider a user running a DISCWorld client (DWc) program on his WWW browser which is connected into a set of DISCWorld servers each running the DISCWorld daemon (DWd). The user has already logged onto the system by asking one of the servers to act as his gateway and has indicated he wishes to browse a collection of satellite imagery with a view to determining weather conditions over a certain period of time. The first step involves the user downloading a satellite catalogue browsing applet [15] which runs in his DWc environment. Through the services it offers he is able to pose queries in terms of particular spatial area, times and dates and finds the sequence of satellite images that will indicate cloud cover over his region of interest. These images may be very large data files which are not directly useful for the user to download to his local computer. Not only would they require too much bandwidth to download,
Remote Data Access in Distributed OO Middleware
4
but the processing capabilities of his computer are inadequate for the job at hand. Instead the catalogue browser returns him a set of references or DRAM’s that act as pointers to the remote data he has just found. The DRAM’s that the user now has may be manipulated in two different ways. Within his DWc environment he may manipulate them locally, as graphical icons. In the context of satellite images, it will be useful to provide small thumbnail image samplers that provide a very low resolution representation of the large images to which they point. If he does wish to download a particular image for detailed inspection, a client applet may be provided to let him do so, perhaps with multiresolution or tiling capabilities built in. More typically the DRAM is just used as a way of specifying the remote data to be used as input to a subsequent computation. In this example a set of images might be fed to a cloud recognition program which might produce cloud masks for the spatial areas of interest. These bit mask results might also be returned to the user as DRAM’s, which he might then feed onto a statistical averaging service which computes spatial and time averages of cloud cover. This reduced data might also be returned in the form of a DRAM. In this case however the data may be relatively lightweight in size and the DRAM representation is encoded to actually carry the data as a cached copy inside it. DRAM’s therefore provide a useful way to encode objects for transmission between hosts depending upon whether they represent large or small data sets. We discuss the performance consequences of manipulating remote data in this way in section 6. We have prototyped the idea of DRAM’s using pure Java code in our experimental DISCWorld system, but we believe the model for DRAM’s can be applied more generally. We can circumvent the dangling pointer problem for DRAM’s because of the nature of the DISCWorld system. The bulk data archives [4,14] at which DISCWorld is targeted at contain relatively slowly changing data so we can state in advance that a server is unlikely to discard a bulk data item between references made to it by remote DRAM’s. The nature of the data products that DISCWorld manipulates is such that the DRAM can carry a descriptive recipe of how to reconstruct the data product even if it is no longer where the DRAM pointed to. This ability to regenerate data products on demand is a key concept behind DISCWorld and is described in [19]. In summary, DRAMs are named in an architecture-independent and locationindependent manner. Object names are constructed from a serialised representation of both the services and data used to create the object. This is possible since services and data are themselves named in a location-independent manner. Services and data can be replicated many times across the distributed system. Data is created within the DISCWorld in a single-assignment fashion. Once data is created, any modification will result in new data being created, with a different global name, derived from the original data product name [12]. The main objective of DRAM’s is to allow client programs to retain a token, or pointer, to a remote data entity. The token may be also be used as an input to further processing and allows different application components running on the client to exchange remote data references. This is illustrated in figure 3. DRAM’s may either be evaluated at the client side and the data they point to downloaded to the client, or may be used in a subsequent instruction to the server to further process that data, or indeed to pass the data to another server, which itself acts as a client to the original server. DRAM’s act as a proxy for the data to which they refer, but in the context of DISCWorld do not suffer from the dangling pointer problems that a simple implementation would incur. The remote object that DRAM’s refer to can either be a data entity or an operation or service that embodies some code that can be executed on a DISCWorld host. For the purposes of this paper we focus on portable Java byte-codes that
Remote Data Access in Distributed OO Middleware
5
can be executed on any Java Virtual Machine (JVM), but we have considered the implications of managing inventories of more general codes [18]. There are three primary scenarios in which data-pointing DRAM’s provide a useful vehicle for data exchange. 1. a client receives a DRAM from a server (and may do further lightweight processing on it locally) 2. a client requests DRAM be used with a service on the same server 3. client requests DRAM be sent to a remote server (a three-way transaction) Scenario number 1 requires that the client is able to call in the DRAM option and actually download (a cached copy of ) the data to which the DRAM points. Scenario number 2 is useful when the client is issuing a series of interactive instructions to a remote server to deal with large data which need never be downloaded to the client. This is useful if the client is an interactive one connected into the server infrastructure by a low bandwidth link. Scenario number 2 builds on this remote data manipulation idea and is useful when cooperating servers which are connected to one another by a broadband network are available to the client which is not so well connected. We describe the object structure and operations we have implemented to enable these three scenarios in section 4. It is also useful to use DRAM’s as a vehicle for transporting portable codes or services between DISCWorld nodes. We have subclassed our DRAM design to be capable of service transport too and have experimented with use of relational databases of code objects that can be made available as a service itself [18].
3
DRAM Notation
We use the following notation when describing the specification and behaviour of DRAM’s: • d represents a DRAM, • D represents data, • S represents a server (or client). This is shown in figure 2. upon which we base the following description of DRAM’s.
d
DISCWorld Remote Archive Mechanism (DRAM)
D
Data Object
DISCWorld Server
S Figure 2: Symbols used to explain DRAM’s.
DRAM’s are defined to point to data, not other DRAM’s. This therefore limits the number of indirection levels to one and reflects our design goal that DRAM’s be pointers to either bulk data products or code services. The notation of di → di+1 → D signifies that both di and di+1 point to D, not each other, and that one may have been created by copying, or cloning, the other. d{i} → Dj means d{i} each point to Dj ∀i
(1)
Remote Data Access in Distributed OO Middleware
6
If the data, D, that a DRAM, d, points to is discarded by the server, by virtue of the scheme whereby the system’s name for the data uniquely describes how to make it, the data can be recreated by the server if it cannot be located using a resource discovery mechanism described in section 2. di → dj → Dk => di = dj
(2)
It is possible to have DAGs of di ’s. There is an n : 1 relationship between d and D. d1 → d2 → · · · → dj → · · · → dn → Dx
Server 1
dj
=> d1 → Dx => d2 → Dx .. . => dn → Dx
Server 2 DRAM d j
"points to"
dk
"points to"
DRAM d k (could be a copy of dj )
Di (Bulk) Data item D i Server 3 DISCWorld Remote Access Mechanims (DRAMs) DRAMs are more than pointers, they also contain data descriptions or recipes. Figure 3: DISCWorld Remote Access Mechanism (DRAM) allows servers to point to remotely stored bulk data items, and to copy these pseudo pointers to other servers. DRAM’s are more than pointers, since they contain a description of how a DISCWorld server could (re)construct the bulk data item if necessary.
The most important DRAM concept is embodied in figure 3. A set of three DISCWorld nodes, servers as shown, cooperate in some calculation. Server 1 has a DRAM dj that points to a bulk data entity Di located under the control of Server 3. Server 2 has been given a DRAM that also points to this data entity and it may have either been set up as an independent transaction between servers 2 and 3 or it may have been passed a copy of Server 1’s dj . In either case both can refer to the bulk data entity and can use it as input to remote computations. The data
Remote Data Access in Distributed OO Middleware
7
belongs to Server 3 and only Server 3 can set the actual policy for deletion of the data entity. Either or both of servers 1 and 2 can delete their DRAM reference to it however. Typically the data entity will be a long lived data product that may be archived to a long term store according to Server 3’s policy specifications, but is unlikely to be deleted outright. Alternatively it may be a derived data product that Server 3 knows how to recreate locally should it decide it is more cost effective to do so rather than store it.
d
D
D
S
S
S
Before
i)
S
After
d
d
D
D
d’ S
S’
S
Before
ii)
S’
After
d
d
D
D
d’
d’ S
iii)
S
Before
S
S
After
Figure 4: i) Creating a DRAM, ii) Copying a DRAM, and, iii) Moving a DRAM
There are a number of operations that can be defined for DRAM’s. We summarise these symbolically in table 1. Figure 4 shows the consequences of DRAM creation, copying and moving. These are the most important operations. Create This operation is the most fundamental operation that can be performed on a DRAM. Once the DRAM object is instantiated, the act of invoking its create method causes it to be bound to a data object, as shown in figure 4 i). This will typically be done as one operation using the DRAM constructor. Copy Copying a DRAM causes its contents to be copied into another instance of the DRAM object. This operation is necessary to allow the propagation of DRAM’s around the DISCWorld system when a node wishes to retain a pointer to the remote data. This is shown graphically in figure 4 ii). Move This operation provides the mechanism for a DRAM to move between servers in the DISCWorld system, as is illustrated in figure 4 iii). It is by this mechanism that DRAM’s are transferred between DISCWorld servers. This is
Remote Data Access in Distributed OO Middleware Operation create copy move copy data capture data discard inspect
8
Definition CR:: D → (D, d) CP:: d → (d, d0 ) MV:: ds → ds0 COPY DATA:: (ds , Ds0 ) → (ds , Ds , Ds0 ) CAPTURE DATA:: (d, Ds ) → (d, Ds0 ) DISCARD:: ds → ∅ INSPECT:: (ds , Ds ) → (ds , Ds )
Table 1: Formal definitions of operations on DRAM’s, where each definition is in the form of before→after and s, s0 are servers
implemented using the Java Remote Method Invocation (RMI) mechanisms in our present implementation. Copy Data This operation causes a DRAM to download (a copy of) the data to which it refers from the remote server. Copy Data is most often used by clients before inspecting data, and by servers to create a local copy before using it in a computation. Capture Data Capturing the data that a DRAM refers to is the equivalent of moving the data from the remote server to the local node. The node effectively now owns the data entity and no longer retains a reference to it on the remote server. However, it is a matter of policy for the remote sever to decide whether it will actually delete its retained copy of the original data entity. Discard Causes the DRAM to be completely destroyed. Note that this does not affect the original data entity, local or remote, to which the DRAM points, as data is stored and managed independently. Inspect Inspecting a DRAM causes the local data to be displayed in the user’s workspace using whatever default method is defined for the data type. For example, in the case of a DRAM representing an image, the default action would be to display it in a simple image viewer applet, whereas if the DRAM refers to a byte-code, the service’s GUI would be displayed on the workspace. It is part of the DRAM creation process to instantiate the DRAM reference locally before passing it to a remote node. It is therefore possible for a node to have a number of DRAM’s pointing to its own local data. We rely largely on constraint properties of the bulk data entities (long lived-ness and read-mostly) managed under DISCWorld as well as explicit data management policies set by server administrators to avoid the need for incorporating call-back into DRAM’s. A server issuing DRAM’s to its data does not retain knowledge, under our present implementation, of where the DRAM’s went. We have considered implementing a special class of DRAM for which call-back is implemented however, as a vehicle for experiments in distributed garbage collection.
4
DRAM Implementation
Our current implementation of DRAM’s is written completely in Java [7] version 2.0. DRAM’s are represented by a superclass DRAM which is subclassed into two domain-specific classes, DRAMS representing DISCWorld services and DRAMD, which represents data in the DISCWorld system.
Remote Data Access in Distributed OO Middleware
9
We describe the DISCWorld metacomputing environment is a client-serverserver architecture, meaning that in extension of the customary client-server system, servers are also able to act as clients to other servers in a totally symmetric manner. Clients and server support codes are both written in Java; the prototype implementation comprises approximately 7,000 lines of code, and while the graphical client is currently restricted to running on Windows NT workstations due to limitations in the Java 2.0 implementations presently available, the server daemons are entirely platform independent within the scope of availability of the Java Virtual Machine (and the Java runtime environment). DISCWorld server daemons host a number of services, predominantly written in Java at present, although we have built some supercomputer codes that are invoked as native methods from within the Java framework [18]. Application services may make Java native method calls to legacy or high-performance programs. Upon the user connecting and authenticating [8] themselves with the DISCWorld system, the server is queried as to which services and data it has available or has information about. Servers exchange information about known services, servers and data via a “gossip protocol” [20], which allows the servers to maintain up to date lists of popular services. The server creates DRAM’s (either DRAMS’ or DRAMD’s for services and data, respectively) and returns them to the client, where they are cached. Figure 5 shows an interactive client screen as would be run from inside a WWW browser. (We have used the Java AppletViewer to make the screendumps for clarity). A user will have logged on to the DISCWorld environment, using one of the participating server nodes as a gateway, and as shown has downloaded imagery catalogue browser applets which can guide him through the data search process. The screendump shows that the user has already found two images which match his search criteria, and these have been returned to him as DRAM’s. These appear as thumbnail iconic forms of the found images. We have associated various mouse operation semantics with DRAM’s. In the case of retrieved images, the available operations allow the user to download the full image for visual display on his client workspace, or inspection of a textual description of the data the DRAM points to, or most commonly, its use as a remote token to another remote computation. The user then has the option of either using pre-written “meta-services” which represent a process network of services, or designing their own process network. The situation where the user chooses a pre-written meta-service is shown in figure 5, where the user is asked to enter data describing a satellite image that they wish to inspect. Results (DRAM’s or strictly speaking DRAMD’s) are placed onto the workspace of the user client, and users have the ability to invoke further operations on them or inspect the data. The only methods that are associated with the DRAM are the operations, as detailed in table 1. The inspect operation, being data type-dependent, is handled by the workspace. It is the workspace’s responsibility to load the correct classes/methods to inspect the data in a meaningful way. We implement DRAM’s as object instances of a DRAM superclass or its subclasses DRAMD and DRAMS. For each of the object variables shown below, there exist get and set methods in the style of Java Beans [5]. The method definitions have been omitted for brevity in the core API definition below. public abstract class DRAM implements Serializable { private String publicName; // descriptive name for Users’ use private String globalName; // internal ID private Icon icon; // associated icon eg thumbnail image private String description; // long free textual description private String className; // query-able search-able text
Remote Data Access in Distributed OO Middleware
10
Figure 5: Earth Observation Information Catalogue (ERIC) applets downloaded into a DISCWorld graphical client environment, running as a Java applet in a WWW browser environment. The ERIC applet provides search forms to identify satellite imagery stored on a remote server, and the two iconic DRAM’s at the lower left are remote data pointers (DRAM’s) to found data.
private String remoteServer;
// representation of class // location of the Data to which // the DRAM points
} public class DRAMD extends DRAM implements Serializable { private Object localObject; // the local (cached) copy (if any) private DRAMS []operations; // allowable operations on the object } public class DRAMS extends DRAM implements Serializable { private DRAMP []inputParameters; // parameters for the service private DRAMP []outputParameters; private Class localClass; // Bytecode for the portable // service }
Remote Data Access in Distributed OO Middleware
11
Figure 6: The user has invoked the download operation on one of the found DRAM’s and has downloaded a copy of the data to which it points onto his display screen area. The data is now cached on the client and can be manipulated more speedily locally, The DRAM pointer to the server copy may still exist however, if the user has not explicitly discarded it.
The object variables contained within the DRAM class are the fundamental building blocks from which a remote data access mechanism can be built. The public name is the descriptive name which can be displayed to users in inventories or in graphical user environments. The name is meaningless to the DISCWorld system, per se, but exists to aide the user in selecting appropriate data items and services from a large collection of such items. The global name is the full name of the data object used by the DISCWorld system to uniquely identify a data item or service. Global names, and their generation, are discussed more fully in section 5. An icon is included in the basic DRAM class so that when used in a graphical environment, the user is able to see a pictorial representation of the data that this DRAM represents, whether it is a generic icon as may be used to represent a string or an integer, or whether it is a thumbnail representation of an image. A description field is provided such that the user is able to either provide a textual description (or annotation) of the data that they have created, or read a description that is already attached to a DRAM. We envisage that DRAMS’ will provide descriptions for the DRAMD’s that are created as products of execution, and that users will have the ability to attach customised descriptions or annotations
Remote Data Access in Distributed OO Middleware
12
which will be retained when the DRAM is sent between clients or servers in the DISCWorld environment. The description will, in conjunction with the public and global names, provide keywords for a search mechanism so that the user is able to find services or data with descriptive data containing certain textual characteristics. The class name is a textual representation of the class that this DRAM represents. It is included so that the user is easily able to query the DRAM and also for use, especially in graphical environments so that the input and output types of services in process networks match. A textual representation of the remote server is retained so that the system knows where the data that the DRAM refers to resides. If a client or server copies or captures the data that the DRAM refers to, which has the effect of copying the data to the local server, the reference to the remote server is updated to reflect the local server. As such, when the DRAM is passed to another client or server, it is provided with the most up-to-date information as to where the data can be found. This information is included with each DRAM to avoid the case where a client, which has not copied or captured the DRAM’s remote data, passes the DRAM to a remote server and several request messages must be sent between the remote client and remote servers. In our model, the receiving client or server is made explicitly aware of the last known location of the data or service, and it is able to request it from there. In the case that a DRAM points to a data object, a DRAMD is created which extends the DRAM class by the addition of a variable which is able to store the data object when it is copied or captured by the DRAM. In addition to storing the object that the DRAMD points to, the DRAM is extended to include a number of operations that are common to DRAMD’s of its type. For example, for imagery operations, the array may contain references to services which provide cropping, histogram equalization, and image convolution operations [9]. Instance Variable publicName globalName icon description className remoteServer localObject operations
Value GMS5 98122500:IR1 ERIC:SingleImage:GMSImage(IR1,00,25,12,98,0,2291,0,2291) thumbnailed representation of data GMS5 image 00Hrs 25DEC98, IR1 channel, Original Dimensions image.satellite.GMS5 cairngorm.cs.adelaide.edu.au:1965 null array containing references to crop, histogram equalise and image convolve operations
Table 2: Typical values for instance variables in a DRAMD For example, a DRAMD to represent a GMS5 satellite image, which is categorised by the date and time of creation and spectral channel has typical instance variables and values shown in table 2. If the DRAM represents a service, a DRAMS object is created which again extends the DRAM class by providing a reference to the downloaded class bytecode when it is copied or captured from a remote server. A DRAMS object also has embedded in it an array of input and an array of output parameters that are associated with the service. This allows, in addition to the manipulation of remote pointers to the service, the assignment of parameters to an instance of the remote service, and the submission of the process network to the DISCWorld system for processing. An example of a DRAMS, which corresponds
Remote Data Access in Distributed OO Middleware Instance Variable publicName globalName icon description className remoteServer inputParameters
outputParameters localClass
13
Typical Value ERIC Retrieve Single Image Service Service:ERIC:SingleImage assigned by creator of service ERIC Service for Retrieving Single Image from GMS5 Satellite Repository satellite.GMS5.ERIC.SingleImage cairngorm.cs.adelaide.edu.au:1965 array containing DRAMD’s to represent integers for the day, month, year, time and scale of image to be returned, as well as the area of interest, and a string representing the spectral channel array containing a DRAMD of type image.satellite.GMS5 null
Table 3: Typical values for instance variables in a DRAMS
to the ERIC service as shown in figure 5 is given in table 3. We have made heavy use of string representations in our DRAM/DISCWorld implementation. This is convenient as it allows for easy free textual searches, and easy interoperability with other OO systems such as CORBA [1,16] and Java DBM [17] interfaces to relational databases.
5
DRAM Naming
DRAM’s are named so as to be location-independent. When searching for a DRAM that is to be executed or used as input to a service, the server is encouraged to first check that the data does not reside in a local cache, but in the event of it not being available, the remoteServer variable contains the last known location of the data. DRAM’s are named according to their data type, creating service, and the input parameters that were used by the creating service. The only exceptions to this naming rule is where fundamental types, such as integers are used as inputs to services, for example, defining a bounding box in an imagery crop service. In our current implementation, to reduce the data as a consequence of passing around references to integers, the name given to such DRAM’s is of the form Integer:x where x is the value of the integer. We have not fully established the best type system space to use for DISCWorld, and indeed this is an interesting research topic in its own right. For our present experimental system however, we believe we have provided a useful prototype space for experimenting with DRAM’s. DRAMS’ are created by a method which resides within the service byte-code. Each service has its own scheme by which it distinguishes between the DRAMD’s that are supplied as inputs, and those that are supplied as outputs. For example, for the above-mentioned DRAMS in the ERIC example, the input parameters may be named DAY, MONTH, YEAR, SCALE, CHANNEL, AREA START X, AREA END X, AREA START Y and AREA END Y. When using the DRAMS in a query, a number of DRAMD’s which represent the inputs to the service must be specified. In the current implementation, only the DRAMD name is attached to the DRAMS when it is submitted to
Remote Data Access in Distributed OO Middleware
14
the DISCWorld server for processing. This is to take advantage of the fact that quite often the server that is doing the processing will be the one that supplied the DRAM to the client (or remote server) in the first place. Output parameters are referred to by exactly the same mechanism: they are given private names by which the service names the outputs. In the case of the ERIC example DRAMS, there in only a single output, called GMSImage. The output data is named in accordance with the input parameters that were used to create the data, coupled with the service name that created it, and the service’s private name for discerning the output from any other. For example, if the DRAMS above were used, the DRAMD shown in the ERIC example may be created, with the appropriate globalName. It can be seen that the globalName is comprised of the service that created the data (“ERIC:SingleImage”), and the private name given to the output parameter by the service (“GMSImage”), as well as the input parameters used to create the data (“(IR1,00,25,12,98,0,2291,0,2291)”). If this DRAMD were to be used as an input to a subsequent service, the globalName of the data would be assigned to the input parameter of the service, and the output from the subsequent service would contain the whole name of the DRAMD as part of the output name. Servers can discard and create DRAMs to local data. This has the effect of updating the remote name contained in the DRAM to point to the most recent data location. We believe that in principle, this naming mechanism is sound, certainly for our present experimental system. We are still experimenting with how namespace should be apportioned to independent DISCWorld clusters and service creators [17]. We believe it is possible to set up an evolving namespace that incorporates Internet protocol (IP) addresses or author email addresses as unique global keys that can be used to guarantee uniqueness of names.
6
DRAM Performance Model and Behaviour
We now discuss the performance consequences of using DRAM’s to mediate data between remote computations on DISCWorld servers. Consider the performance consequences of the example given in section 2. The following sequence of transactions occur. 1. User downloads client applet and formulates query 2. Applet invokes catalogue search on Server 1 which retrieves the data from deep store, perhaps caching them on disk if they were stored on tape. 3. DRAM’s pointing to the retrieved data is returned to the user 4. The user invokes a processing service (cloud cover calculation) on server 2, which is closely connected to server 1 (either they share disk mounts or at least are connected by a broadband network) 5. Server 2 completes its task, and returns DRAM’s pointing to the results (cloud masks) to the client user. 6. User now invokes a statistical data reduction service on the sequence of cloud masks and this runs on server 3. 7. server 3 now returns a DRAM pointing to the result, and if this is small according to some local policy set by the administrator of server 3, the DRAM may contain a cached copy of the actual data.
Remote Data Access in Distributed OO Middleware
15
At each stage, the DRAM’s returned to the client may be bundled with lightweight service codes to allow its manipulation. These codes may be cached at the client host, so that once it already has a copy of the necessary code, there is no longer any need to download it again. This will work properly providing each service has a unique global name. Typical values for the various data sizes and transactions costs for our archive of Geostationary Satellite Imagery archive [15] are given in table 4 Applet upload/startup time Typical Raw Image Size Typical Modem Connectivity Inter-Server Connectivity Adelaide/Canberra Latency Cloud Cover Mask for SA
3-5 seconds 5 Mbytes (GMS5) 7 kbytes/sec (56kbps) 16 Mbytes/sec (ATM OC-3) 20 msec 64 kbytes
Table 4: Data sizes and Transmission rates for Cloud Cover Example Posing the query involves interaction between the user and client. It is faster if an intelligent client program can be run locally and can catch typing mistakes and other inappropriate entries by the user before the query is committed. High latencies between the client and the server make it preferable if the number of exchange transactions is minimised. Our experimental system between the Australian cities of Adelaide and Canberra involves a typical latency of around 20ms, but we have also used remote computer systems in Japan, which are approximately 200ms latency distant. This can considerably slow the query formulation if a X-terminal approach were used and all the programs were run remote from the user. Suppose the query results in a sequence of 100 images to be retrieved from the primary store on server 1. This 500Mbytes of data would take around 20 hours to download to the client over a typical modem line. The time taken for two cooperating servers to exchange the data would only be around 30 seconds over a typical ATM based OC-3 line. This is a typical access time we measure for our archive, but would be increased to between 5 and 10 minutes if the data had to be retrieved from tape store first. The processing time involved in the cloud cover calculation is relatively low for computers with large enough memory that the entire data set can be manipulated in core. Typical geostationary images are a few megabytes in size and can be processed in a few minutes on present computer platforms [15]. Processing time is likely to be at most a few minutes for computing cloud cover even for a sequence of 100 images. The time will be dominated by disk access times. For this sort of application scenario we conclude the DRAM notion is useful since it: • allows the user to operate remotely; • avoids ever having to download full sized datasets to the client; • allows the client to work with services which may be running on separate hosts; • allows for deferred delivery of the final reduced decision support data product to the client without the need to keep a connection open to the servers. More generally the DRAM notion and the DISCWorld framework allows query based computations of this sort which would not be feasible in terms of transaction time or practicality otherwise. We have measured approximate times based on a clients and remote servers based at widely geographically separate locations in
Remote Data Access in Distributed OO Middleware
16
Adelaide, Canberra and Japan, with clients operating over low bandwidth local or modem links. Our prototype system does work and aside from vagaries of different Java Virtual Machines does show the expected performance.
7
Discussion
It is also convenient to use DRAM’s as a mechanism for deferred delivery of computations. A DRAM may be delivered to a client program with an indication that the remote data to which it refers will only be available within certain times. This allows the token to be redeemed at some time in the future when a long computation has completed. The concept of a deferred delivery mechanism is also useful where a user or client program receives a token to some data that may not yet be created or directly available. This allows the user or client to submit a request, and upon a subsequent login to the DISCWorld environment, receive the results. The token is used both as a placeholder for the request result, but also as proof of authorisation to request the status of the pending request. DRAM’s are used to implement the deferred delivery mechanism of DISCWorld, and are the equivalent of data tokens. Common mechanisms for notifying users of deferred delivery products in other systems include the creation of output files in pre-defined directories, and the receipt of electronic mail bearing an authentication key to be used to gain access to the results at a server. We believe that while these methods are indeed efficient and perform well for their intended applications, the DISCWorld philosophy is to abstract away from the user’s file system and to use a more scalable, portable mechanisms such as embodied by DRAM’s. Although our original concept for a DRAM was that of a remote data pointer in an object-oriented framework. It was convenient to extend this idea to allow DRAM’s to have a special graphical or iconic representation in a DISCWorld client environment. Our present implementation allows for a DRAM to be attached to a graphical icon in the client screen area, and to be manipulated using the usual mouse semantics. These include being able to drag and drop iconic DRAM’s onto other applications and other sensitive screen areas. We are presently exploring collaborating graphical clients that allow two interactive users to share a graphical workspace and to exchange data using DRAM’s. Graphical User Interface builder environments such as AVS and JavaStudio make use of similar graphical semantics to move around iconic representations of code and data items and allow these to be interconnected. Our DISCWorld service infrastructure lends itself particularly well to such semantics and we are also building this capability into the DISCWorld graphical client environment. DRAM’s have a direct analogy to the way in which user queries are submitted to the DISCWorld metacomputing environment. As shown in figure 5, Service DRAM’s and Data DRAM’s are able to be combined together into a process network, which depicts the manner in which a result is constructed. The process network is then either serialised, and sent to the DISCWorld server for processing, or is converted into a textual form before being sent to the DISCWorld server.
7.1
Comparison with other Systems
The remote data access problem is not readily implemented by existing mechanisms in network-oriented programming systems such as Java’s networking package or RPC. In such systems, all results are returned to the user upon completion of their request. What is needed, and is not supported by either Java or RPC, is a method whereby the user is returned a pointer to a result, that is able to be accessed directly
Remote Data Access in Distributed OO Middleware
17
if they wish to inspect the data that the pointer represents, or is able to be passed to another server for further processing. DRAM’s provide for this need. The problem of global addresses for pointers is tackled by the construction and use of canonical names for data. This is possible under the general DISCWorld framework. There is no concept of the pointer referring to physical address space on any particular machine in the distributed infrastructure; we believe this enables DRAM’s to be scalable and realisable. The Nexus run-time system [6] utilises a similar concept, called the Remote Service Request (RSR), to provide a global pointer space in the language that runs above Nexus, Compositional C++. The RSR mechanism is far lower-level than the DRAM, as it only encapsulates the data-source machine address and memory location of the data it is referring to. Useful information such as the size and type of data being referred to is not passed. Nexus RSR’s do share some similarities with DRAM’s in that when the RSR is dereferenced, or the DRAM inspected, the remote data is able to be retrieved, and both RSR’s and DRAM’s are able to be copied, moved and sent between any client and servers without requiring the data to be retrieved. Thus it is possible, in principle, to construct a DRAM implementation using the Nexus run-time system. DRAM objects implement a lot more than the RSR simple token mechanism does. Microsoft has Object Linking and Embedding (OLE) technology built into its operating systems. This provides the ability for the user to pass references to objects for the purposes of embedding within other objects, for example allowing a spreadsheet to be embedded within a word-processing document. Unfortunately there is no concept of being able to remotely process the data that the object represents. DRAM’s, in addition to representing remote data, allow the user to specify operations to be performed on the data, possibly yielding new products, which may occur remotely, according to the client-server-server architecture of DISCWorld. In addition, the data that the DRAM represents becomes independent of the program that was used to create it, as opposed to the situation in OLE in which the program that created the data (or an equivalent program) must be used to view the data. Chandy’s Infospheres [2] project has some similar objectives to that of DISCWorld. This project attempts to address dynamic scalable distributed systems by composing components. In the Infospheres context, components are processes and sessions, where sessions are processes composed in parallel. Asynchronous RPC is used for interprocess communication. Processes send messages to other processes requesting that they change their state. Processes have incoming and outgoing message queues with output queues being able to be bound to input queues. The problem of deferred delivery mechanisms and data abstraction is avoided by creating pipes that run between processes, which do not allow data to be re-used between computations. We believe there is some commonality between the Djinn mechanism used by InfoSpheres and our DRAM’s, although we believe DRAM’s provide a more compact solution to the data reference transport problem. Table 5 summarises some key comparisons between our DRAM mechanism and the Internet Inter-Orb Protocol (IIOP) and the Interoperable Object Reference (IOR) mechanisms in CORBA [1]. We were inspired by the use of remote references as a form of pointer in CORBA for our development of DRAM’s and indeed DRAM’s can be implemented using a combination of IIOP and IOR. However, DRAM’s provide some novel and we believe vital extra functionality for a robust distributed system. DRAM’s, IIOP’s and IOR’s all point to remote objects. IIOP’s are object references that are used to point to objects on remote Object Request Brokers (ORBs). IIOP’s are intended for referencing objects within an ORB’s administrative boundary. IIOP’s consist of an ORB reference and the memory location of the object within that ORB. In contrast, IOR’s are used for referencing objects across ORB
Remote Data Access in Distributed OO Middleware
Characteristic granularity
DRAM’s high level (refer to remote object by name)
evaluation strategy
can be used for demand-driven computation
lifetime and robustness metadata
DRAM outlives an instance of the DWd (can not dangle)
equivalency
Equivalency of DRAM names enables optimisation of data access between servers
DRAM describes the object being referenced (type, size description, runtime, etc)
18
CORBA IIOP/IOR IIOP is low level (refer to remote object by global address IOR is high level, refers to object by name method called when IIOP dereferenced object created when IOR dereferenced IIOP’s valid for single instance of ORB (refers to a memory address) IOR independent of object instance IIOP is just a remote pointer (contains type info and transport protocols) IOR can contain miscellaneous data – totally implementation dependent IIOP is just a remote pointer (pointers are perhaps only equivalent if they point to the same object on the same ORB) IOR equivalence through naming
Table 5: Comparison between DRAM’s and CORBA’s Interoperable Object References (IOR’s) and Internet Inter-Orb Protocol (IIOP)
boundaries. IOR’s are adapters that specify the protocol with which a remote object can be referenced. An IOR provides a high-level, ORB-independent representation of a server object, for use across ORB boundaries. The user of an IOR is not aware of the ORB from which the IOR originates. When an IOR’s object is dereferenced, it is created on demand at the ORB from which the IOR came. Thus, in some ways, the IOR is similar to the DISCWorld Global Naming Mechanism, where objects are able to be referenced by name, rather than address, on a remote server. DRAM’s are a high-level construct, referencing an object on a DISCWorld server by name, as opposed to an IIOP, which references objects by a global address. On restart of a DISCWorld daemon or an ORB, a DRAM reference would still be valid, while the IIOP would not, since a CORBA object would almost certainly be created with a different memory address in the ORB. DRAM’s are independent of memory addresses. When a DRAM is used to construct a complex processing request, the object to which the DRAM refers is not actually used until it is demanded by the DISCWorld daemon. CORBA IIOP does not allow for on-demand or “lazy” processing. Original CORBA objects are dereferenced as soon as their references are used. This functionality is assumed by the IOR, with which references can be traded between machines. To aid construction of processing requests, DRAM’s contain metadata describing the objects to which they refer, as we describe in section 4. The CORBA IOR mandates type information only. IOR’s can carry miscellaneous data on the object they represent, but this is user/application dependent. The DRAM definition mandates sufficient metadata or “recipe” information that will allow reconstruction of
Remote Data Access in Distributed OO Middleware
19
the pointed-to object in case of a restart. The DRAM mechanism allows a remote node the option of transferring the object, to which a DRAM refers, to the local DISCWorld node. A CORBA object reference is simply a reference to the remote object. Mechanisms to transfer objects must be supplied explicitly by the CORBA user/application. Using a global naming mechanism to refer to DRAM’s, we allow equivalency between DRAM’s to be established. This naming scheme is independent of the DRAM mechanism and we assume its existence as does CORBA. We can use a simple implementation of the unique naming scheme provided by either Java/RMI or by CORBA. These are usually based on “mangled” strings derived from host names or URLs. IIOP’s are only equivalent if they point to the same object on the same ORB, whereas DRAM’s from different nodes can have the same name, implying that they are copies of the same object. In conclusion, we believe that DRAM’s provide a more scalable approach to remote data manipulation than does CORBA IIOP/IOR alone. It is however possible to implement a DRAM-like middleware mechanism using IIOP and IOR as an underlying technology instead of the Java/RMI underlying technology we have used. However, although DRAM’s could be implemented in CORBA/C++ (ie without Java), the functionality would be somewhat restricted. All processing would have to be performed at the server-side since the compiled object code would no longer be platform independent. Since a DRAM is a Java object, it can have extra functionality programmed in for use at the client end. This could be graphical semantics (see section 4) or data filter/compression services. For example, data to be transferred could be compressed at the server and uncompressed at the client end explicitly by the DRAM framework. This would not be possible without the bytecode portability provided by Java. In so far as simple inter-node referencing is concerned we find no significant difference in performance of IOR’s over DRAM’s. We believe the added functionality and robustness of DRAM’s over the existing CORBA mechanisms are crucial to a wide area distributed metacomputing system.
8
Summary and Conclusions
We have presented our ideas for an object-oriented data transfer mechanism to be used in a metacomputing middleware. Our DRAM draws on ideas extant in other systems such as ORBs and Java remote method invocation. We have developed a formal notation for further investigation of the behaviour of such pointer objects and have developed a basic implementation of the data transfer DRAM. DRAM’s allow for non-RPC style or “connectionless” or non-blocking communications. DRAM’s mandate sufficient metadata to allow re-establishment of a broken or deferred connection, or re-creation of discarded or garbage-collected remote objects. DRAM’s allow mobile data and code. Although the implementation of DRAM’s is experimental, we believe that the model and notation and experimental implementation provides us with a basis for further research in a number of areas. Of particular interest at present is the area of namespace management especially for independent DISCWorld user communities who are independently assembling collections of data repositories and associated services. We believe the string mangling mechanisms of both CORBA and Java/RMI can be improved upon. We also believe there is scope for work in a distributed garbage collection and management system using DRAM’s as an implementation vehicle. We are working on the use of DRAM’s as connective tokens in a pipelining
Remote Data Access in Distributed OO Middleware
20
scripting language to be used in DISCWorld and also on iconic DRAM’s in client builder environments for graphical process networks. Our initial experiments with DRAM’s do yield the expected performance benefits and provide a good data transport mechanism model.
Acknowledgments Thanks to J.F.Hercus for assistance in designing the graphical client components described in this paper. This work was carried out under the Distributed High Performance Computing Infrastructure Project (DHPC-I) [21] of the On-Line Data Archives Program (OLDA) of the Advanced Computational Systems (ACSys) Cooperative Research Centre (CRC) and funded by the Research Data Networks (RDN) CRC. ACSys and RDN are funded by the Australian Commonwealth Government CRC Program.
References [1] Object Management Group, CORBA/IIOP 2.2 Specification, July, 1998, Available from http://www.omg.org/corba/cichpter.html. [2] K. Mani Chandy, Adam Rifkin, Paulo A.G./Sivilotti, Jacob Mandelson, Matthew Richardson, Wesley Tanaka and Luke Weisman, A Wide area system using Java and the Internet, Proc. HPDC 1996. [3] P.D.Coddington, K.A.Hawick and H.A.James, Web-Based Access to Distributed High-Performance Geographic Information Systems for Decision Support, To Appear in Proc. of Hawai’i International Conference on System Sciences (HICSS-32), Maui, January 1999. [4] P.D. Coddington, K.A. Hawick, Et Al., Implementation of a Geospatial Imagery Digital Library using Java and CORBA, Proc. Technologies of ObjectOriented Languages and Systems Asia (TOOLS 27), Beijing, Sept 1998, J. Chen et al., (IEEE, 1998) . [5] Robert Englander, Developing Java Beans, Pub. O’Reilly, ISBN 1-56592289-1, 1997. [6] Ian Foster and Carl Kesselman, Globus: A Meta-computing Infrastructure Toolkit, http://www.globus.org, 1996. [7] James Gosling, Bill Joy, Guy Steele, The Java Language Specification, Pub. Addison Wesley, The Java Series, 1996, ISBN 0-201-63451-1. [8] Duncan A. Grove, Andrew J. Silis, J.A.Mathew and K.A.Hawick, Secure Transmission of Portable Code Objects in a Metacomputing Environment, Proc. PDPTA, Las Vegas, July 1998, also Technical Note DHPC-041. [9] K.A.Hawick and H.A.James Distributed High-Performance Computation for Remote Sensing Proc. of Supercomputing ’97, San Jose, November 1997 [10] K.A.Hawick et al., The DISCWorld Metacomputing Environment, Invited article for special issue of Int. J. Future Generation Computing Systems, on Metacomputing, Editor. W.Gentzsch. [11] K.A.Hawick, H.A.James, C.J.Patten and F.A.Vaughan, DISCWorld: A Distributed High Performance Computing Environment, Proc. of HPCN Europe ’98, Amsterdam, April 1998.
Remote Data Access in Distributed OO Middleware
21
[12] Heath A. James, Scheduling in Metacomputing Systems, PhD Thesis, The University of Adelaide, July 1999. [13] H.A.James and K.A.Hawick, Resource Descriptions for Job Scheduling in DISCWorld, Proc 5th IDEA Workshop, Fremantle, Feb 1998, Also Technical Report DHPC-026. [14] K.A.Hawick and P.D.Coddington, Interfacing to Distributed Active Data Archives, To appear in International Journal on Future Generation Computer Systems, Special Issue on Interfacing to Scientific Data Archives, Editor, Roy Williams. [15] H.A.James and K.A.Hawick, A Web-based Interface for On-Demand Processing of Satellite Imagery Archives, Proc. ACSC’98, Perth, February 1998, Also Technical Report DHPC-018. [16] K.E.Kerry and K.A.Hawick, Service Management in DISCWorld using CORBA, Proc 5th IDEA Workshop, Fremantle, Feb 1998, Also Technical Report DHPC-033. [17] J.A.Mathew and K.A.Hawick, Querying and Auxiliary Data in the DISCWorld, Proc 5th IDEA Workshop, Fremantle, Feb 1998, Also Technical Report DHPC-030. [18] J.A.Mathew, A.J.Silis and K.A.Hawick, Inter Server Transport of Java Byte Code in a Metacomputing Environment, Proc. TOOLS Pacific (Tools 28) - Technology of Object-Oriented Languages and Systems, Melbourne, 1998. [19] C.J.Patten, F.A.Vaughan, K.A.Hawick and A.L.Brown, DWorFS: File System Support for Legacy Applications in DISCWorld, Proc 5th IDEA Workshop, Fremantle, Feb 1998, Also Technical Report DHPC-032. [20] Andrew Silis and K.A.Hawick, The DISCWorld Peer-To-Peer Architecture, Proc 5th IDEA Workshop, Fremantle, Feb 1998, Also Technical Report DHPC-028. [21] Distributed & High http://dhpc.adelaide.edu.au/.
Performance
Computing
Group
