An Architecture for A Scalable Wide Area Distributed System Philip Homburg, Maarten van Steen, Andrew S. Tanenbaum Vrije Universiteit, Amsterdam fphilip,steen,
[email protected] Abstract Current wide area distributed systems fail to hide implementation aspects related to the distribution of information. Unfortunately, the solutions developed for local systems do not scale, so a different approach is needed. In this paper we present an architecture for distributed systems that scales to support a large number of users and objects. The architecture is based on the concept of distributed shared objects, which are used for sharing and exchanging information over wide area networks. Our architecture provides location and replication transparency, and offers the flexibility needed to cope with the heterogeneity inherent in wide area systems.
1 Introduction Distributed systems provide sharing of resources and information over a computer network. A key design issue that makes these systems attractive is that all aspects related to distribution are transparent to users. In other words, a distributed system should appear to be an integrated collection of services implemented on a single computer. The attractiveness of this approach is illustrated by the World Wide Web [1]. Although many improvements are still needed to achieve full transparency, accessing a Web page requires, in principle, only the selection of a hyperlink. 1
Aspects regarding the location of a page, accessing the site where it is stored, and transferring it to the user are hidden by the system. Unfortunately, general-purpose wide area distributed systems that allow users to share and manage arbitrary resources in a transparent way hardly exist. Instead, transparency in distributed systems has only been achieved for local systems. As it turns out, the solutions for achieving transparency developed for local systems do not scale to the wide area case. In particular, they generally do not take into account the most important properties that characterize wide area systems:
A very large number of users and resources. An inherent latency problem caused by the distance between nodes. Heterogeneity due to a variety of underlying operating systems and networks. Involvement of multiple administrative organizations. The goal of the research described in this paper is the design and implementation of a wide area distributed system that provides transparency. We anticipate that our system will be constructed as a middleware solution on top of existing operating systems and networks [2]. It will facilitate the straightforward development of wide area applications. An application developer need not be concerned with the distribution and replication of data and the communication across the underlying network. In addition, our system will be capable of supporting a very large number of users and resources. As a first step towards such a system, we describe an architecture that can incorporate scalable solutions. The solutions themselves are not discussed in this paper. Instead, we discuss an architecture offering a framework to construct scalable wide area systems in which different solutions can be incorporated simultaneously. However, we do show that existing architectures for distributed systems are inherently restricted to support scalability, and are thus not suited for the wide area case. The paper is organized as follows. In Section 2 we describe the main problems that need to be solved. A proposal for an architecture for a scalable wide area distributed system
2
that solves these problems, is described in Section 3. We conclude with a discussion and comparison to related work.
2 Problems to be Solved In this section we concentrate on four key problems that need to be solved by wide area distributed systems: a uniform model for information exchange, location and replication transparency, flexibility with respect to implementations, and scalability.
2.1 A Uniform Model Wide area applications provide users facilities for sharing and exchanging information. To that end, numerous abstract communication models have been adopted. For example, the World Wide Web presents its users with a view of a distributed document consisting of pages that are tied together through hyperlinks. Electronic mail services offer a model in which messages are deposited in mailboxes. News applications allow users to subscribe to news groups, and read and post articles. Finally, systems such as Jade [15] and AFS [17] offer traditional file operations in a wide area context. All of these applications have a simple, uniform conceptual model behind them. These models are generally implemented by using low-level communication primitives. A large variety of such primitives exist: RPC, point-to-point message-passing, stream-oriented communication, distributed shared memory, etc. However, these low-level communication models are a source of problems. First, they do not provide solutions to communication problems that are common to all wide area systems, such as replication and data migration. The effect is that each application has to invent an ad hoc solution. Second, the variety of facilities leads to different implementations, making it difficult to integrate applications into a single framework or to exchange information between applications. Finally, the low-level communication models are generally not standardized with respect to semantics and interfaces. Consequently, developers are faced with an addi-
3
tional heterogeneity problem for which they have to adopt solutions as part of their application. What is needed is a uniform communication model to express sharing and exchange of information. The model should be more abstract than those offered by the simple communication primitives but still be application-independent. It should incorporate aspects common to all wide area applications, and allow abstract models to be implemented in a straightforward way.
2.2 Location and Replication Transparency Another problem with current wide area applications is the lack of transparency they offer with respect to the location(s) of data. In practice, a name is tightly coupled to the location of the object it refers to. There are, however, several drawbacks to this approach. For exampl e, if a person moves to another company, his or her personal Web page will have to be renamed. For example, http://www.abc.com/jan
becomes http://www.xyz.com/jan.
An obvious drawback is that location-dependent names make it difficult to deal with migration. If a name refers to where data is located, migrating that data cannot be done without either changing the name or giving it another interpretation. For example, a name may be interpreted as a forwarding pointer to the actual location of the data. In wide area systems, this may lead to long chains of forwarding references, which are not only inefficient but also highly susceptible to node and link failures. Another serious drawback is that location-dependent names cannot be used straightforwardly when referring to replicated data. Effectively, users will be forced to keep track of which data is replicated, where the replicas are located, and how consistency between these replicas is maintained. However, replication and consistency schemes are generally so difficult that they cannot be handled easily by users [3]. They should therefore be hidden entirely. The problem that needs to be solved is that a wide area system should offer a naming facility that hides all aspects concerning the location of data. Users should not be concerned where data is located, whether it can move, whether it is replicated, and if it is replicated, how consistency between replicas is maintained. This mechanism should be available to all applications as a standard facility.
4
2.3 Flexibility In contrast to local distributed systems, wide area systems have a hard requirement for support of flexibility with respect to implementations. In particular, wide area distributed systems need to provide services with unambiguous functional semantics, but with flexible consistency semantics. Interfaces to these services should be independent of their implementation. These requirements follow from the fact that heterogeneity in wide area systems is unavoidable. Heterogeneity here has several faces. First, there is the variety in services, functionality, and interfaces of the underlying operating systems and networks that need to be dealt with. Second, global state is generally distributed in a non uniform way across the system. Combined with differences in latency, bandwidth, and failure semantics in different parts of the system, the accessibility of global state as perceived by clients will vary as well. The result is that we need flexibility with respect to how we implement the functionality and the consistency of services. In particular, completely different implementations for the same interface may need to co-exist. These implementations all realize the same functionality, but may differ with respect to state consistency. For example, a client that needs up-to-date information may require an implementation that incorporates replicated state which is kept sequentially consistent with other replicas. On the other hand, for a less demanding client, it may suffice to install a cache that is not immediately refreshed on every update of the global state.
2.4 Scalability Finally, the most important problem which needs to be solved is that of scalability. For one thing, the rapid growth of the user population on the Internet, suggests that a general-purpose wide area distributed system should be capable of supporting millions of users. It is conceivable that a system may eventually have to support one billion users, each having thousands of files. The problem of scalability is enormous and no single solution will exist [12]. In order to manage the complexity of the scalability problem, we believe it is essential to first devise an architecture in which different solutions can be embedded. Such an architecture is the subject of this paper and is presented in the next section. 5
Application Middleware Operating system
Network
Figure 1: The position of the middleware layer.
3 Proposed Architecture In this section we discuss the main components of an architecture for a scalable wide area distributed system, and argue that it can support a very large number of users and resources.
3.1 An Object-based Approach The problems described in the previous section are not new: they have been addressed and solved for local distributed systems. Solutions have been embedded in what is called middleware [2]. Middleware forms a layer on top of operating systems and network services and provides a uniform, platform-independent interface as illustrated in Figure 1. It serves two purposes: (1) it offers a model for information exchange that is common to all applications, and (2) it achieves transparency by hiding implementation aspects related to the distribution of resources. We also propose a middleware solution. An important question then is which model for information exchange should be offered. A promising approach that has also been adopted by others is to provide an object-based model. The encapsulation of state and its operations into a single object allows a clear separation between the services provided by an object and their implementations. Aspects such as communication protocols, replication strategies, and distribution and migration of state can be completely hidden behind an object’s interface. The concept of an object offers us the transparency and flexibility needed for wide area systems. 6
Object shared by processes A and B
Address space of process A on machine #1
Address space of process B on machine #2
Object "/uk/co/www/index"
Object "/nl/vu/cs/home/philip/mbox"
Object "/edu/mit/lcs/info"
Object currently not in any address space
Object currently used only by process A
Figure 2: Processes can share objects by having them mapped into their address spaces at the same time.
7
However, most object-based models assume a client/server model of computation. This means that an object is managed by a server that accepts method invocations from remote clients. This model, however, does not scale to the wide area case. What is needed instead is a symmetric model of computation, as illustrated in Figure 2 (and which is further explained below). Processes cooperate through distributed shared objects, objects of which the state is distributed and shared between processes. There is no a priori distinction between clients and servers. Furthermore, we propose passive objects: objects provide state and methods but no activity. Activity is only provided by processes (or threads). Changes to the state made by one process are observed by other processes. The combined effect of distributed shared state and passive objects is a model in which information is shared and exchanged in a simple and straightforward way. Our general model of distributed objects is organized into an architecture which consists of three major components:
A process model giving the application programmer’s view of distributed objects. An architecture describing the organization of a distributed object. Infrastructure and system services to support those distributed objects. These three components are discussed below.
3.2 A Process Model We describe many aspects of our model as operations on address spaces. An address space is a segment of (virtual) memory. Address spaces are part of processes and of operating system kernels. Address spaces are used to make a distinction between two kinds of objects. A local object is restricted to exactly one address space and forms the building block for distributed objects. Distributed objects are objects that can be mapped into multiple address spaces at the same time. They are implemented as a collection of local objects and appear to their users as a single, global shared object. A distributed object has methods for operating on its internal state. These methods are exported via interfaces. Our model supports the separation of (compiled) application programs from 8
object implementations. A program and an object implementation have only an interface in common. At runtime, a process can load one of the implementations of an object into its address space. The object can be automatically shared with other processes all over the world in a fully transparent way. Before a process can use a distributed object, it has to bind to that object. Names are used to allow binding. During the binding, code and data needed to support the object are loaded into the process’ address space. Multiple processes can share an object by binding to it simultaneously. The code that is loaded during binding is contained in a class object, which represents a class of objects with the same implementation. A class object is a local object which contains the implementation of the methods of such objects. It is used to instantiate a new local object representing the distributed object. Our process model provides the following features to support binding to distributed objects, and to support the separation between a program and an implementation of a distributed object: 1. Access to a distributed object is provided by a local object which represents the distributed object. This isolates the program which uses a distributed object from the implementation of the distributed object. The local object can provide the same interface, independent of the communication or replication protocols that are used. This contrasts, for example, RPCbased systems which have a one-to-one mapping between the interface presented to the user of an object and requests sent over the network. 2. The runtime system supports downloading of new objects into the address space of a running process. Both state and code are downloaded dynamically on demand. 3. Method invocation is done through interface tables, each listing one or more methods. Interface tables decouple the application from the implementation of the local object. An object may export several interfaces. Multiple interfaces are useful in two different ways: they can be used to group functionally related methods, and they can be used to support multiple versions of an interface, each having slightly different semantics. An interface table has one entry per method. Each entry in the table contains a method pointer and a state pointer. The method pointer points to the executable code which imple9
ments the method. The state pointer points to the object’s state. Storing a method pointer separate from a state pointer allows multiple objects to share the same code, and allows a single interface to hold methods from different objects. A single interface table with methods from different objects can be used to implement composite objects: objects that are aggregations of other objects. 4. The local object which represents the distributed object is conceptually self-contained. This means that the implementation of a distributed object is completely separated from the application. The only part that the application and the object share is the layout of the interface table and the syntax and semantics of the methods in that interface table. Finally, our process model includes objects that encapsulate threads, locks, memory allocators, etc. Together with communication objects, these allow for object implementations which are independent of the underlying operating system. Furthermore, applications use objects through interface tables which are defined in a language-independent way. This allows applications written in one language to use objects written in another.
3.3 Distributed Objects The architecture for a distributed object consist of two parts. The first part consists of local objects, one per address space, which communicate across a network. This part interacts with the process model and application interface presented above. The second part of the architecture, and the part we focus on here, describes the internals of the local objects in each address space. This distinction is illustrated in Figure 3. This part of the architecture is designed to be extremely flexible. The operations that can be performed locally and the communication protocol that is used are completely encapsulated within the local object. Distributed objects that provide the same interface to the user can have different local implementations. For example, at one extreme the local object may be an RPC client stub which forwards all requests to a server. At the other end of the spectrum we have fully replicated state which is updated using totally ordered group communication.
10
Distributed object A2
A1
Address space
Network Entry object
Local object
Replication object
User object
Communication object
A3
A4
Control object
A5
Replication object
Communication object
Figure 3: The structure of a distributed object.
11
Semantics object
To simplify the implementation of new distributed objects we propose a composite local object that consists of four (sub)objects. Figure 3 shows these four objects: a communication object handling low-level communication with other local objects; a replication object dealing with state distribution (and the consistency of that state); a semantics object providing the actual semantics of the distributed object; and a control object handling local concurrency control, and calling the replication object and the semantics object in the right order. These four objects can be divided into two groups: the communication object and the replication object are expected to be generic objects that deal with (replicated) state without attaching any semantics to it. They are bound in from the appropriate library. The other group, the semantics object and the control object are specific to a distributed object: they provide an interpretation of its state and provide the methods. The semantics object is written by the user and does the real work. We anticipate that the control object is mechanically generated by a compiler from a specification of the semantics object (analogous to the generation of client and server stubs in the case of RPC). Communication objects provide a simple interface to send data. A distinction is made between communication objects for point-to-point (connection-oriented and connectionless) data transfer and for multicasting. A communication object is used by a replication object to keep the state of the distributed object consistent. The replication object deals with two kinds of data: marshalled copies of (parts of) the object’s state, and marshalled arguments and return values of operations. Marshalled state and marshalled arguments are provided by the control object. For example, if a replication object acts as an RPC stub, it does nothing more than send marshalled arguments to the remote server and wait for the results. On the server side, the marshalled requests are passed to the control object and marshalled requests are sent back. On the other hand, a primary-backup replication object may pass around new copies of the state of the object from the primary to the backups after a write operation has been performed. Likewise, a replication object that supports active replication only has to send marshalled arguments around. Note that the replication object decides whether and when any state is stored in the semantics object. The replication object can, for example, make this decision based on observed read/write ratios. 12
The semantics object and the control object deal with the user-defined aspects of the distributed object. The semantics object maintains one copy of the state of the object and provides the operations on the state. The control object is expected to marshall and unmarshall state, arguments, and return values. It also knows which operations modify the state of the semantics object, and which do not. This information is passed to the replication object.
3.4 Support Services In this subsection we describe the support services needed for a process to bind itself to a distributed object. The support services for persistence, security, caching and highly replicated objects are beyond the scope of this paper and are the subject of ongoing research. A distributed object provides a set of protocols that can be used by an address space to participate in the object’s implementation. Address spaces can participate in different ways: it is possible to get a copy of the state of the distributed object and execute operations locally. Alternatively, operation code and parameters can be forwarded to another address space so the operations can be executed there. In this context, a participation protocol includes both the lower layers of communication protocols that send bits across networks and higher layers that deal with replication protocols and data encoding. For each participation protocol, the distributed object offers a set of contact addresses. Multiple contact addresses per protocol improve fault-tolerance and are needed to benefit from replication. A participation protocol is represented by a worldwide unique identifier. How a distributed object can be contacted is stored as (protocol-ID, address) pairs, also called contact points. As described in the previous sections, the local object provides access to the distributed object. The local object is installed and initialized during the bind process. The bind process maps a name to a class identifier for the local object and a contact point that is to be used by the local object to contact the distributed object. The class identifier is used to load a class object which is then used to create a new instance: the local object. Next, the local object is initialized with the contact address. The implementation of the binding process uses four services as shown in Figure 4: a name
13
service, a location service, a destination selection service, and an implementation repository. The name service and the location service are worldwide and together they map a user chosen name for an object to (protocol-ID, address) pairs that can be used to contact the object. The destination selection service and the implementation repository are local support services that provide local information and implement local policies. Object name
Name lookup
Destination selection
Object handle
Object’s contact point
Location lookup
Implementation selection
List of object’s contact points
Class object
Figure 4: Services used during the binding process. During the bind process, the object’s name is first looked up by the name service. Object names are maintained by users and applications, independent of the location or storage of the object. A name lookup yields an object handle, which is used as an index in the location service. The location service maps this index to the set of contact points for the object. This separation of naming and location supports local and replication transparency: if an object moves, or in general, if the set of contact points changes, the object informs the location service about the change. Users can rename objects or create aliases for objects without affecting the objects themselves. The set of contact points returned by the location services is handed to a local destination selection service. This service maintains information about remote destinations, whether they are reachable or not, expected latency and expected bandwidth. The destination selection service may also maintain some information about the network topology, administrative policies and pricing information. This information is used to order the set of contact addresses returned by the location service. Protocol identifiers are then used by the implementation repository to select a suitable implementation of a local object that “speaks” the protocol. If no implementations are available for a given protocol identifier, the implementation repository can try to locate such an implementation or the contact point has to be discarded. 14
While this four-step lookup process may seem heavyweight, it provides the flexibility essential to a huge worldwide system. The name service and the location service are internally structured as a collection of directory objects. The directory objects of the name service map a component of a name to an object handle. This handle can refer to any object, including a directory object. Together, these directory objects form a directed graph, with one directory object appointed as the root. The location service is structured similarly, except that the directory objects in the location service map bit strings to (protocol-ID, address) pairs. This organization allows the name service and location service to benefit from the support for replication present in distributed objects. The state of popular directory objects, for example those close to the root, can be highly replicated. Contact addresses for highly replicated objects can be replicated as well. In addition to support for replication, this scheme also allows delegation of parts of the name space to different organizations in a straightforward way.
3.5 Scalability A scalable architecture for distributed objects has to solve two problems: how to manage an extremely large number of objects, and how to deal with objects that have a huge number of users. The scalability of our design depends on the scalability of the distributed objects and of the support services. Distributed objects.
The scalability issues related to distributed objects are critical to our ar-
chitecture. Although we do anticipate that a very large number of objects can be supported, it is reasonable to assume that relatively few objects are actively used at any time. The number of these objects influences the scalability of our architecture. In particular, with many actively used objects, use of communication resources may become a bottleneck. Efficient usage of these resources can be realized through clever implementations of communication objects, for example by multiplexing existing unicast and multicast channels. The point to note is that our architecture provides the right place to encapsulate such implementations and hide them from other parts of
15
a distributed object. In contrast, existing systems do not support such improvements. For example, the HTTP protocol as used by the World Wide Web, sets up one TCP connection per access request. Scalable solutions for objects with many users should be centered around hierarchies. General solutions, that is, independent of the semantics of a particular object, incorporate hierarchy as part of either communication objects or replication objects. When incorporated in communication objects, a replication object will perceive communication as taking place within a large group with many members. Such an organization is feasible if the number of state updates is very low or if updates originate from a small, stable set of group members. For example, the equivalent of a radio station in our architecture would have one group member that generates updates, and many readers. On the other hand, hierarchical organizations incorporated in replication objects are useful when updates are relatively frequent and come from different sources. In that case, it is essential that efficient update schemes be used, which implies that replication objects should optimize on communication. This can be done if communication between the different parts of a distributed object is explicitly structured in communication graphs. Support services. The organization of the name service and the location service into directory objects allows us to deal with a large number of objects. These directory objects will be hierarchically organized and effectively partition the total state of either service. In addition, each directory object is designed and implemented as a distributed object. The scalability of the support services is thus dependent on the scalability of distributed objects. For example, it can be expected that the state of the directory objects close to the root will be highly replicated and distributed, and possibly even hierarchically organized as described above. Again, it is important to note that although we do not provide specific solutions to scalability problems in this paper, our architecture allows several kinds of solutions to be incorporated. This contrasts many existing models which, by their nature, cannot scale to wide area systems. In other words, we provide a scalable architecture.
16
4 Discussion and Related Work In this paper we have presented an architecture for a scalable wide area distributed system. Our approach has in common with others that it adopts a middleware solution. However, in comparing middleware solutions, differences exist with respect to interfaces and organization. Interfaces are either expressed in terms of a programming language, or through special interface libraries. Language-based interfaces, such as offered by Network Objects [5] have the benefit that they generally support application development at a relatively high level of abstraction. A more restricted use of languages is offered by systems such as Fragmented Objects [11] and Thor [10] in which new object types are to be specified in a special definition language. However, we feel that a language-based interface should only be used for convenience. General-purpose wide area systems should be completely independent of programming languages to be of practical use. Instead, they should allow applications to be written in any language, and if necessary, in different languages. The organization of the middleware generally focuses on providing either convenient communication primitives, or is otherwise based on some object model. Communication-based solutions are provided, for example, by systems such as ISIS and Horus which put an emphasis on reliable group communication [4], or DCE which is centered around client/server computing based on RPC [16]. These approaches are not suited to solve the problems of large-scale wide area distributed systems. Although many problems are related to scalable communication, it is not the only problem that needs to be solved. In particular, communication-based solutions tend to ignore the problems related to flexibility with respect to implementations as we have discussed in this paper. The most promising approach also followed by many others seems to be that of an object model implemented as part of the middleware. Popular object models are those in which the state of an object is always contained in a single address space. Clients access that state through a proxy that offers the same interface as the object and which is responsible for marshalling and communication with the object. These, what we call centralized object models have been adopted by CORBA [13], Clouds [7], COOL [9], SOR [18], and others. The World Wide Web offers a 17
similar model: each page can be viewed as an object with a single fetch operation. Centralized object models are, however, inadequate for general-purpose wide area systems. First, using proxies essentially implies an RPC-based solution which scales poorly as it is sensitive to latencies [14]. Second, these models require distribution and replication schemes that, by their definition, cannot be encapsulated in a single object – the models lack plurality transparency [6]. A separate mechanism must therefore be provided to replace or adapt middleware policies. This not only violates the idea behind object-orientation, it also restricts flexibility. Our approach differs from others in the sense that an object’s state may be distributed and replicated across multiple address spaces. Distribution and replication schemes are encapsulated by an object: they form part of its implementation. In other words, we provide truly distributed objects. Similar to our approach is the model offered by the Spring system [8]. Their subcontracts more or less correspond to our replication and communication objects. However, the concept of subcontracts is based on a client/server model of computation, and processes are to make direct use of them. In contrast, we offer a symmetric model of computation: we make no distinction between clients and servers beforehand. Instead, a process participates in the implementation of a distributed object, exemplified by incorporating a semantics object in a process’ address space. Not only is our approach more flexible, it also provides better support for object-specific organization, replication and distribution of state. This is crucial with respect to scalability. The research described in this paper is still in an early stage, and many problems, for example security and persistence, have yet to be fully addressed. Likewise, compositional techniques are currently being developed in order to adequately support very large scale objects. These, and other topics are subjects of ongoing research.
References [1] T. Berners-Lee, R. Cailliau, H. Frystyk Nielson, and A. Secret. “The World-Wide Web.” Comm. of the ACM, 37(8):76–82, Aug. 1994.
18
[2] P. Bernstein. “Middleware, An Architecture for Distributed System Services.” Technical Report CRL 93.6, Digital Equipment Corporation, Cambridge Research Lab, Mar. 1993. [3] K. Birman. “The Process Group Approach to Reliable Distributed Computing.” Comm. of the ACM, 36(12):36–53, Dec. 1993. [4] K. Birman and R. van Renesse, (eds.). Reliable Distributed Computing with the Isis Toolkit. IEEE Computer Society Press, Los Alamitos, CA, 1994. [5] A. Birrell, G. Nelson, S. Owicki, and E. Wobber. “Network Objects.” In Proc. 14th Symp. on Operating Systems Principles, pp. 217–230, Asheville, NC, Dec. 1993. ACM. [6] A. Black and M. Immel. “Encapsulating Plurality.” In O. Nierstrasz, (ed.), Proc. 7th European Conference on Object-Oriented Programming, volume 707 of Lecture Notes in Computer Science, pp. 56–79. Springer-Verlag, Berlin, 1993. [7] P. Dasgupta, R. LeBlanc, M. Ahamad, and U. Ramachandran. “The Clouds Distributed Operating System.” Computer, 24(11):34–44, Nov. 1991. [8] G. Hamilton, M. Powell, and J. Mitchell. “Subcontract: A Flexible Base for Distributed Programming.” In Proc. 14th Symp. on Operating Systems Principles, Asheville, NC, Dec. 1993. ACM. [9] R. Lea, C. Jacquemont, and E. Phillevesse. “COOL: System Support for Distributed Programming.” Comm. of the ACM, 36(9):37–46, Sep. 1993. [10] B. Liskov, M. Day, and L. Shrira. “Distributed Object Management in Thor.” In T. Ozsu, U. Dayal, and P. Valduriez, (eds.), Distributed Object Management, pp. 79–91. Morgan Kaufman, Los Altos, CA, 1993. [11] M. Makpangou, Y. Gourhant, J.-P. Le Narzul, and M. Shapiro. “Structuring Distributed Applications as Fragmented Objects.” Technical Report 1404, INRIA, Jan. 1991. [12] B. Neuman. “Scale in Distributed Systems.” In T. Casavant and M. Singhal, (eds.), Readings in Distributed Computing Systems, pp. 463–489. IEEE Computer Society Press, Los Alamitos, CA, 1994. [13] Object Management Group. “The Common Object Request Broker: Architecture and Specification, version 1.2.” Technical Report 93.12.43, OMG, Dec. 1993. [14] C. Partridge. Gigabit Networking. Addison-Wesley, Reading, MA, 1994. 19
[15] H.C. Rao and L.L. Peterson. “Accessing Files in an Internet: The Jade File System.” IEEE Trans. on Software Eng., 19(6):613–624, Jun. 1993. [16] W. Rosenberry, D. Kenney, and G. Fisher. Understanding DCE. O’Reilly & Associates, Sebastopol, CA, 1992. [17] M. Satyanarayanan. “Scalable, Secure, and Highly Available Distributed File Access.” Computer, 23(5):9–21, May. 1990. [18] M. Shapiro. “Flexible Bindings for Fine-Grain, Distributed Objects.” Technical Report 2007, INRIA, Aug. 1993.
20
