Each base level actor has a meta level actor, called messenger, which serves as the customized ... messenger for an individual message. In other .... level 0 msg.
Design and Implementation of a Safe, Reflective Middleware Framework Nalini Venkatasubramanian, Sebastian Gutierrez-Nolasco, Shivjit Mohapatra & Jehan Wickramasuriya Dept. of Information & Computer Science University of California, Irvine Irvine, CA 92697-3425, USA {nalini,seguti,mopy,jwickram}@ics.uci.edu
Abstract With the rapid evolution of the global information infrastructure and ubiquitous computing environments, service providers will need to provide effective and adaptive resource management mechanisms that can serve concurrent applications in the presence of changing system conditions. Flexible, scalable and customizable middleware can be used as enabling technology for next generation systems that can satisfy application requirements (real-time, security, mobility, fault tolerance etc.) executing in highly dynamic distributed environments. To enable application aware resource management, we are developing a customizable and safe middleware framework called CompOSE|Q based on a reflective metamodel. In this paper, we motivate the need for safe composability of concurrently executing middleware that can interact and interfere with each other. We describe the architecture and runtime environment for CompOSE|Q and assess the performance overhead of the additional flexibility. We also illustrate how application requirements such as security and timeliness can be supported efficiently in the CompOSE|Q framework. Keywords: Actors, Distributed systems, Meta-Architecture, Middleware, Reflection, Composability.
1
Introduction
In the near future, large scale distributed systems must handle applications that are open and interactive. Applications over the digital wireless Internet such as smart houses, intelligent transportation systems, mobile information appliances such as palmtops running Web-based multimedia applications require dynamic reconfiguration of systems, services and protocols. 1
Open Distributed Systems (ODS) allow services, end-systems and protocols to dynamically attach to and detach from the distributed environment; such systems evolve dynamically and their components interact with an environment that is not under their control. Increasing levels of distribution in information access, openness and dynamism results in increasing complexity. Middleware is the fundamental infrastructure that enables distributed computing. Flexible middleware platforms incorporate the notion of reflection in order to provide the desired level of configurability and openness in a controlled manner [35]. Yet, there exist several challenges in delivering a safe plug-and-play middleware environment where protocols and services are installed and uninstalled on the fly to accommodate dynamicity. Firstly, distributed applications have varying requirements, often stated as QoS (Quality of Service) parameters that define the extent to which performance specifications such as responsiveness, reliability, availability, cost-effective utilization and security may be violated. These application requirements can be satisfied using appropriate resource management policies, e.g. replication, migration, checkpointing. Satisfying application QoS in a dynamic environment implies dynamically customizing the underlying management mechanisms to meet QoS requirements. A prime issue in the execution of middleware entities is that of safe composability. During the concurrent execution of multiple distributed resource management services and protocols, complications arise in the dynamic customization of these services and protocols. Sometimes the interactions can be complex and rapidly changing, causing violations in correctness and alteration of the semantics of applications built above the generic distributed management infrastructures. Much of this has been ignored in existing commercial frameworks and distributed middleware infrastructures. Frameworks such as CORBA and DCOM represent a step toward compositional software architectures but do not deal with interactions of multiple object services executing at the same time, and the implication of composing object services. Building a QoS-enabled customizable middleware framework requires characterizing and reasoning about the interactions between multiple resource management activities in ODSs, their dynamic installation and customization in the presence of QoS requirements. In this paper, we describe the design and implementation a QoS-enabled reflective middleware framework called CompOSE|Q currently being developed at the University of California, Irvine. CompOSE|Q is based on a meta-architectural model that facilitates specifying and 2
reasoning about the composability of multiple resource management services in ODSs. We illustrate how effective mechanisms can be incorporated into a composable middleware framework to ensure safety and QoS enforcement in distributed environments. The rest of this paper is organized as follows. We begin by briefly describing the TLAM, a formal semantic model for reasoning about middleware in Section 2. Section 3 discusses the basic CompOSE|Q architecture and metalevel services. Section 4 addresses the implementation of the CompOSE|Q system and presents initial performance results. Related work is discussed in Section 5. We conclude in Section 6 with future research directions.
2
The Two Level Meta Architectural Model (TLAM): A Formal Model for Reflective Middleware
To simplify development of applications, we use Actors [1], a model of concurrent active objects that has a built-in notion of encapsulation and interaction among the concurrent components of an ODS. In the actor paradigm, the universe contains computational agents called actors, distributed over a network. Traditional passive objects encapsulate state and a set of procedures that manipulate the state; actors extend this by encapsulating a thread of control as well. Each actor potentially executes in parallel with other actors and may communicate with other actors via asynchronous message passing. Using Actors, we define a meta-architecture framework that permits customization of resource management mechanisms such as placement, scheduling and synchronization. Ensuring correctness in a purely reflective model involves reasoning about system level interactions by characterizing the semantics of shared distributed resources and understanding what correctness of the overall system means. The TLAM (Two Level Actor Machine) model [46] is a first step towards providing a formal semantics for specifying and reasoning about properties of and interactions between components of ODSs. In the TLAM, a system is composed of two kinds of actors, base actors and meta actors, distributed over a network of processing nodes. Base level actors carry out application level computation, while meta-actors are part of the runtime system, which manages system resources and controls the runtime behavior of the base level. Meta-actors communicate with each other via message passing as do base level actors, 3
but they may also examine and modify the state of the base actors located on the same node. The TLAM uses reification (base object state as data at the meta object level) and reflection (modification of base object state by meta objects) with support for implicit invocation of meta objects in response to changes of base level state. The TLAM provides for full actor-style interaction of meta level objects; we are currently looking into providing restricted support for the explicit invocation of meta objects by base objects. Our earlier work has focused on defining a rigorous mathematical semantics and on using these semantics to develop concepts and methods for expressing and reasoning about properties of middleware components and their interactions. In the TLAM model, meta-level controllers define protocols and mechanisms that customize various aspects of distributed systems management. In practice, multiple system and application activities occur concurrently in a distributed system, e.g. scheduling, protocol processing, stream synchronization etc., and can therefore interfere with each other. Composing multiple resource management mechanisms leads to complex interactions. Consider the following example of a system where distributed garbage collection and process migration can proceed concurrently. If processes in migration (transit) are not accounted for, the garbage collection process can potentially destroy accessible information. Similarly, if the process is continuously migrating, the garbage collection process runs the risk of potential non-termination. In general, risks that arise due to mechanism composition include loss of information, possible non-terminations that cause deadlocks and livelocks, dangling resources, inconsistencies, and incorrect execution semantics. One approach to deal with interference during mechanism composition in an open system is to serialize or delay activities to ensure overall safety of the system. However, this can result in over-serialization of resource management activities causing performance degradation. Furthermore, global delays and halts may cause violations of timing based QoS constraints. We also argue that composability of resource management activities is not just desirable, but essential to ensure cost-effective QoS in distributed systems. For instance, system protocols and activities must not enforce arbitrary delays in the presence of timing based QoS constraints.
4
3
The CompOSE|Q Architecture
Based on the two level meta-architecture, we are developing a customizable and safe distributed systems middleware infrastructure, called CompOSE|Q (Composable Open Software Environment with QoS) that has the ability to provide cost-effective QoS-based distributed resource management. CompOSE|Q provides composable distributed resource management, i.e., it allows the concurrent execution of multiple resource management policies in a distributed system in a safe and correct manner. This will allow safe integration of resource management mechanisms for services such as mobility, load balancing, fault tolerance and end-to-end QoS management. The CompOSE|Q architecture, as illustrated in Fig. 1 contains: • Modules that implement the four basic core services - remote creation, dissemination services, distributed snapshot and directory services with interaction constraints that ensure their concurrent execution with each other and other metalevel services. • Common object services built using core services - actor migration, replication of services and data, actor scheduling, distributed garbage collection, name services, customized communication etc. Each of these services has its own interface definitions and interaction constraints. • Components that provide middleware services - real time, security, fault tolerance and mobility. These components leverage from common object services and allow to customize and adapt their machanisms and policies. • A QoS Brokerage Service enforces (application) QoS requirements by selecting a subset of middleware services, such as security and real time, that provide the required QoS in a safe and correct manner .
3.1
Core Services
To ensure non-interference and manage the complexity of reasoning about components of ODSs in general, our strategy is to identify key system services where non-trivial interactions between the application and system occur, i.e. base-meta interactions. We refer to these key services as core services. Core services are used in specifying and implementing more complex activities within the framework as purely meta-level interactions. The development of suitable non-interference requirements allows us to reason about the composition of multiple system services; these services have constraints that must be obeyed to maintain composability (i.e. 5
Fig. 1: CompOSE|Q high-level architecture.
safe concurrent execution). Examples of such non-interference requirements have been spelled out in [46, 38]. We use commonly observed patterns in distributed systems services to extract implementable abstractions and identify four metalevel core activities: • Remote Creation: Recreation of services/data at a remote site. Remote creation can be used as the basis for designing algorithms for activities such as migration, replication and load balancing. • Distributed Snapshot: Capturing information at multiple nodes/sites used as a basis for checkpointing, distributed garbage collection [41], etc. • Distributed Dissemination: Disseminating information at multiple nodes/sites used as a basis for common services that require different degrees of consistency or synchronization such as replication consistency, clock synchronization, key management, etc. • Directory Services: Interactions with a global repository. Directory services can be used to provide access control, resource discovery and implement group communication protocols. Remote Creation: Remote creation is the process by which actor creation occurs on a specified node other than the node from which creation is being initiated. Remote creation is a basic facility that can be used in other resource management activities like load-balancing, replication and migration. By encapsulating the interactions between the application and system level actors within the remote creation service, we can state requirements that ensure safe and correct composition of other resource management activities with remote creation. In a real 6
TLAM based implementation, the control activities of remote creation are managed by remote creation meta-actors (RCM) residing on every node in the system. A remote creation request has four parameters - a description of the fragment desc to be migrated, the remote node (N), any initial state the desc has to be set to and the initiating-actor β. The initiator actor β is maintained by the RCM to ensure composability with other meta-level services [38]. If the requester needs to know that the request has been completed, or the names of some of the newly created actors, then this can be achieved by specifying appropriate messages as part of the requested fragment, and observing their delivery. Distributed Snapshot Services: Global properties like the number of application-actors, the current reachability graph of distributed actors, number of messages being processed and task queue sizes help in making runtime decisions like load balancing, migration and garbage collection, leading to efficient runtime management of a distributed system. To fully represent the global state of the distributed system, we need a mechanism for recording the state of all nodes including the portion of node state being communicated in the network channels [6]. As state information is accessible explicitly only in nodes, a snapshot mechanism must ensure that node state information in channels are recorded at some node in the system (possibly the target node itself). The snapshot mechanisms we have devised are such that applicationlevel computation and system level services proceed concurrently with the snapshot, thereby preserving application and service semantics. In order to initiate snapshot recording on every node and force messages in channels to reach a node, we have defined two wave protocols for message propagation (described below). The starting and finishing points of both waves, the propagation path and constraints on message propagation are defined elsewhere [38]. Termination is signaled when the wave is complete, serving as a synchronization point in the snapshot mechanism. Note that this generalizes the notions of wave and global snapshot discussed in [12] where the snapshots explicitly do not account for information contained in messages in transit. Distributed Dissemination: Since delivery latencies can vary significantly in a distributed system and some components may require different levels of consistency or synchronization, we have defined mechanisms to propagate information that (1) enforce an adequate level of synchronization (if needed), (2) help to maintain information consistency and (3) coordinate
7
message propagation used by other core services such as the distributed snapshot service. For instance, the wave protocols; (a) visit all nodes exactly once, capturing node-resident information and (b) traverse all links in the system exactly once forcing messages on channels to reach nodes (where their state can be recorded). Directory Services: The challenge here is allowing the middleware to interact with the directory service in a flexible and efficient manner, enabling developers to effectively create a variety of directory-enabled applications quickly and easily. This also helps de-couple context or policy from the application logic, and allows easier handling of interactions between a variety of general purpose services within the middleware (as oppose to analysis on a per-application basis). Directory services play an important role in middleware frameworks, and are utilized in a number of different areas in the CompOSE|Q architecture; (1) resource discovery for load management and admission control; (2) access control information; (3) system information collection and update and other information that may be used by the QoS brokerage service to facilitate the use of various middleware services.
3.2
Common Object Services
Implementation of sophisticated policies and mechanisms for QoS management is made possible by providing support for common services in CompOSE|Q. For instance, object scheduling mechanisms use the basic remote creation core service to assign newly created objects/actors on nodes with adequate resources. Using generalized state capture facilities, we are developing a checkpointing service for capturing causal orders of executions in the system that can be used for monitoring and debugging distributed computations. A state broadcast mechanism is used to implement a clock synchronization service, which informs nodes about a global time value that can be used for time related services. Migration: We now illustrate how a migration service can be built using the remote creation core service. By using RC as the basis for migration, we have ensured composability of migration with other meta-level services [38] such as reachability snapshots and distributed checkpointing. Migration is the process by which actors move from one node to another. The migration service allows for relocation of actors for easier access, availability and load balancing.
8
In this section, we describe a generalized migration service and it’s potential implementation using the remote creation (RC) specification. A migration request is given by a pair (α,ν), where α is the actor to be migrated, and ν is the destination node. This is interpreted as a request to move the computation carried out by α to the node ν. In order to state explicitly invariants maintained by the system during the migration process, we classify the migration process into 3 phases, associating to each phase the configuration in which it begins. The first phase is the initiation phase and specifies the state of the system when the migration request received can be processed. It determines the computation to be migrated by suspending the computation of the actor and noting its current description. In the second configuration the actual actor migration is performed using the RC service. The last configuration finalizes the migration process and establishes transparent access to the migrated actor. Customized Communication: Although middleware platforms provide some notion of QoS specification [51], QoS enforcement [47] and mechanisms to enhance application behavior transparently at run-time [29], they do not deal with flexible and safe composition of communication services. As a result, several (implementation oriented ) communication frameworks are not well suited in distributed and highly dynamic asynchronous environments, where the communication framework must be able to automatically reconfigure itself in order to respond to application requirements and/or changes in the environment by adding, removing or composing communication services dynamically. A communication service is a mechanism that ensures certain desirable properties about the communication between two or more distributed objects, such as security or reliability. Communication services are composed to obtain their combined benefits. However, their service guarantees may crucially depend on the composition order, and a semantic approach to reason about the correct composition of communication services is required. Furthermore, such composition must be constrained in order to prevent functional interference with other services of the system that could lead to an inconsistent configuration state, which may violate the semantics of the basic primitives (safe flexibility). Safe flexibility is required to protect the system from security threats and failures while ensuring high performance; policies to enforce this are often hard-wired across different parts of the system. In order to provide correct composition of communication services to QoS-based applications in a transparent and scalable fashion, while ensuring correctness of basic middleware services in 9
a meta level architecture for distributed resource management (e.g. garbage collection, remote creation), the TLAM model is extended with a composable reflective communication framework (RCF), which customizes the base level communication services among a group of objects as shown in Fig. 2. Each base level actor has a meta level actor, called messenger, which serves as the customized and transparent mail queue for that base level actor. There is one communication manager in every node of the distributed system, which implements and controls the correct composition of communication services specified by the messenger. The messenger has four message queues: the up and down queues are used to communicate with its base level actor, serving as the actor’s send buffer and customized mail queue respectively, while the in and out queues are used for interaction with the communication manager, requesting communication services that satisfy QoS constraints. The up and down queues hold raw messages from and to base level actors, while the out and in queues hold processed messages, which are messages with the required protocols enforced. Furthermore, the communication manager has a set of communication protocol actors, each of them implementing a particular communication service provided by the framework (e.g. reliable protocol, in-order protocol). Communication services can be added (plugged in) or removed (plugged out) dynamically without side effects. The above scheme allows us to abstract a core set of communication services and share it between the different messengers on a node, simplifying the synchronization and composition process, while encouraging separation of concerns in the process of message transmission and reception. In order to maintain accurate semantics and provide an efficient implementation of the architecture, the communication manager implements a set of meta level representatives, called pool-actors. At any instance, the pool-actor handles the communication services requested by a messenger for an individual message. In other words, every message requiring communication services is assigned a pool actor. The pool actor ensures the correct order of composition of required services and provides a coordination mechanism between the messenger that requires the services and the protocols that provide it. This concept of reusable pool-actors is an efficient way to handle the service request of each messenger without having to pay the bottleneck associated with the centralization of the services in the node communication manager.
10
Fig. 2: The RCF model: Each base level actor has a meta level entity; its messenger that interacts with the node communication manager.
3.3
Middleware Services
Middleware provides a standardized way to address interoperability, development, and runtime support for a wide variety of distributed applications in heterogeneous environments. We provide a selection of middleware services as part of CompOSE|Q that allow for safe, customizable operation as well as enabling the implementation of sophisticated policies and mechanisms for resouce management. Here we present two examples of such services we have developed for the CompOSE|Q framework, and illustrate some compositional issues between them from the point of view of the QoS brokerage service. Each of these services addresses a broad area in the middleware framework, but some specific applications are provided as case studies. 3.3.1
Supporting Real-time in CompOSE|Q
Flexible scheduling mechanisms are critical to the design of customizable and reflective middleware frameworks. In future, scheduling services have to be customizable and should be able to dynamically adapt to diverse/interchangeable services (implementing different policies) and dissimilar workloads at different times. Moreover, systems currently deployed in distributed environments (e.g. Windows, UNIX etc.) support best-effort, priority schedulers that provide minimal support for meeting timeliness and dependability requirements of tasks. The growing popularity of soft real-time payloads with quality of service demands makes the design of ef11
fective middleware scheduling services absolutely essential. The overall goals for a distributed scheduler are to achieve predictable and guaranteed operations for individual tasks and provide timeliness and real-time guarantees for real-time applications. In addition, the scheduler should be able to dynamically adapt to changing system and network conditions and improve the overall performance of the entire distributed system. The adaptive scheduler in CompOSE|Q extends the notion of “task scheduling” to include scheduling of tasks within individual nodes (peripheral scheduling) and scheduling of tasks across nodes (focal scheduling). Soft real-time guarantees are met by exploiting the priority based native scheduler, to emulate a real-time scheduler. Furthermore, individual messages are assigned priorities and a conjunctive (task + message) priority scheme is employed to make scheduling decisions. This innovation enables us to fine tune our scheduler to provide both coarse and fine grained scheduling. Any customizations made to the focal scheduler would result in a system-wide change in the scheduling behavior (coarse-grained scheduling); changes made to the peripheral schedulers exploits the capacity and resource availability of individual nodes, providing a more fine-grained control over the scheduling. Case Study: Customizable Scheduling Within the CompOSE|Q framework, actors communicate with each other via asynchronous messaging. A single actor can send and receive messages from any number of independent actors, running either on the same or on different machines. In the absence of our customizable scheduler, the messages are processed in the order of their arrival at the destination actor and all messages are assumed to carry the same importance. To provide a more fine grained notion of priority we introduce priorities into individual messages in the system. The different actors have priorities assigned to them when they start out and the actor priorities are independent of the individual message priorities. The scheduling framework provides application actors methods for setting message priorities and also monitors the assignments so that they conform to the acceptable priority levels. With the knowledge of priorities of the individual messages and the destination actors, we combine the message and task priorities to come up with unique task priority assignments that can be used to optimally schedule tasks (both real-time and non real-time), delivering a stricter notion of priority scheduling. This approach significantly
12
changes the way we provide adaptability and customizations and introduces a new perspective to scheduling in distributed environments. The scheduling framework tries to achieve the overall scheduling goals described above, within a distributed framework that employs active objects and the messaging based communication paradigm. The model is designed as a two-tier architecture, with the following two primary components (see Fig. 3): The Focal Meta Scheduler: The focal scheduler is a system wide global scheduling entity, that is responsible for high level decisions and adaptation that ultimately determine the scheduling characteristics of the entire distributed system. It schedules the application tasks onto the various nodes of the distributed system based on overall load and resource availability. Furthermore, it assigns the various scheduling policies and system wide that invariants that need to be installed on the distributed nodes. The Peripheral Meta Schedulers: The peripheral or local meta schedulers are system level entities that run on the individual nodes (as a part of the node runtime) of the distributed system. They schedule the independent tasks on that node and are also responsible for realizing the conjunctive actor and message scheduling mechanism. The local meta scheduler is logically divided into a two sub-schedulers for scheduling of the real-time and the non real-time tasks respectively. The CompOSE|Q DS forms a communication channel between the focal and the peripheral schedulers. The focal scheduler commits the policy and other decisions into the DS, which is used by the peripheral schedulers. Also, the peripheral schedulers periodically update their state information(system load, current task pool etc.) in the DS as a feedback to the focal scheduler. The peripheral scheduler comprises of a soft realtime sub-scheduler (ASRT: Adaptive Soft Realtime Scheduler) that attempts to provide soft real-time guarantees to such tasks, while still ensuring that the normal time-sharing tasks within the system do not starve. The native operating system task priorities are carefully exploited to enforce the correct order in which the real-time tasks need to be executed. Moreover, a schedule is calculated so that each task gets adequate resources to finish execution within its specified deadline. Both the Earliest Deadline First (EDF) and the Rate Monotonic Scheduling (RMS) algorithms are supported and the 13
RM
D I S T R I B U T E D D S
EDF
Policy Selection
LLF
level 0 msg level 2 msg
UD
Focal Scheduler
Peripheral Scheduler
Monitor Adapt
Monitor Adapt Native Scheduler
ACTOR2
Base+Boost(3)
Re-adjust Priority Base+Boost(1)
Peripheral Scheduler
Monitor Adapt
Increase Priority
ACTOR3
ReceivePot ASRT
ACTOR1
BROKER DISP.
ASRT
Postman Update Sch
Native Scheduler
1. Peek
Aid SM Priority
SCHEDULER
Fig. 4: Conjunctive Scheduling in CompOSE|Q
Fig. 3: Two Level Meta Scheduler in Compose
adaptation module provides for dynamic switching between the above scheduling algorithms based on certain predetermined constraints. The real-time sub scheduler always runs at the highest possible fixed priority and only one real-time task runs at the second highest fixed priority at any time. The scheduler periodically wakes up and dispatches the next the task on the schedule. If there is no real-time task waiting, the scheduler suspends itself letting the native scheduler dispatch one of the timesharing tasks. If there are some real-time tasks waiting to be scheduled, the PMS changes the current active real-time task, from the running state to waiting state, and then dispatches another real-time task from the waiting pool based on the pre-calculated schedule. The various components of the two level meta scheduler are shown in Fig. 3. Both the focal and peripheral schedulers have monitoring and adaptation modules built into them. The monitoring module checks for task compliance. For example, it should be able to detect and stop a rogue real-time task that does not comply with its resource requirement specification and tries to hog system resources. In the context of the global scheduler, the monitoring module systematically observes the transient system load and resource availability. The adaptation module makes use of this information to make intelligent adjustments that could improve the overall system performance. The local adaptation module adapts the peripheral scheduler to changes in the local load and resource availability in a node. Each peripheral scheduler can be logically considered to have two components for scheduling the real-time and the non realtime tasks. The proposed architecture handles real time tasks differently from the time-sharing tasks, and it is therefore reasonable to separate them into different parts.
14
3.3.2
Supporting Security in CompOSE|Q
Security is an obvious concern in distributed environments, however in an object-based architecture where there exists a high degree of concurrency and mobility, the ability to provide fine-grained control over the objects in the system at the middleware layer is desirable. As well as providing security at the application level, access control mechanisms are needed to prevent service and content providers, as well as unauthorized personnel gaining access to client data and resources. The dynamic nature and semantic diversity of object-oriented systems makes it difficult to ensure information security and integrity by the use of traditional access control mechanisms at either the operating systems or application layer. In addition, session-layer security is also an important factor in providing security at the middleware layer. Video streaming is a typical example, where the ability to map user requirements into security mechanisms and have those translate to specific protocols while maintaining QoS requirements is a challenge - particularly with the introduction of real-time constraints. Furthermore, maintaining these security mechanisms across administrative boundaries (or domains) in the presence of mobility is an even greater challenge. In corporate networks, firewalls are utilized to filter out ’unwanted’ traffic from the outside world. However, this type of filtering is generally not powerful enough to detect malicious or anomalous data that can potentially trigger security attacks in OO systems [16]. We utilize this notion of filtering as well as the concept of partitioning the network into a set of federated domains (security domains) in our access control framework. Our general focus is to provide a selection of customizable and safe security solutions at the middleware layer. Mobility induced by the application or the underlying distributed computing environment further complicates the management of access rights within domains. The underlying resource management system can initiate migration of data and computation for load balancing, resource sharing and localized access to information thereby improving the overall performance and reliability of the distributed application. Applications that execute on mobile hosts may hold classified information and execute classified computation while the mobile hosts enter and leave domains with varying levels of security. Thus, a target host (receiver of a message) must verify the access rights of messages received by the transport framework to application components within the (possibly new) host domain. The target host must also ensure that downloaded executable content received in messages (remote programs, applets) do not violate access rights of 15
local data and resources. Similarly the sender of a message must ensure that the recipient of the message is still within a valid domain to receive information and services contained within the message. We present an outline of our access control framework in the context of CompOSE|Q and briefly describe domain-based access control (DBAC). In addition, the system supports techniques for ensuring security in highly mobile environments (both host and object mobility) as well as mechanisms for delegation and revocation of access rights, details of which can be found in [48]. Case Study: An Access Control Framework for Mobile, Concurrent Objects Our reflective model for access control utilizes a meta-level architecture for both enforcement and interpretation of security policy in the system. This meta-level service manages access rights for base actors and ensures that the capablities associated with actors accurately reflect the desired security policies. These management tasks are performed at the meta-level where meta-actors on their respective nodes carry out access control duties, and are coordinated (when necessary) via a global security policy. We model application components as mobile concurrent objects (actors) that are distributed over nodes in the system and provide an access control framework to give us fine-grained control over what these objects can access under varying conditions. Here, we assume the communication layer provides the protocols necessary to establish a secure foundation. In other words, messages cannot be stolen, or tampered with in transit. The primary component in the architecture is the Meta-level Security Actor (MSA), which is resident on each node and is responsible for handling messages destined to actors residing on that node. The global MSA co-ordinates and maintains information that is needed by the local MSAs, and encapsulates global policies for maintaining system-wide security as well as handling the secure bootstrapping of the capability objects (classes). However, local access control enforcement is carried out by MSAs resident on each node, and to a large extent is independent of the global MSA. An outline of the architecture is depicted in Fig. 5. Finally the directory service, which is a trusted store of information; includes the access rights information for each actor in the system, as well as corresponding domain and location information which is pertinent to the MSAs when carrying out enforcement between nodes. The state information corresponding to the access rights for objects in the system can only
16
Fig. 5: The AC Meta-architecture: Interaction between the local MSAs with objects on a node (both Fig. 6: Modelling Domains: Illustrates the interactagging and verification of messages), as well as the tion between various domains in a dynamic mobile coordination duties performed by the global MSA. environment.
be referenced by the MSA; it is typically held within the directory service entry corresponding to the base-actor and cached at the local node MSA. Each local MSA is responsible for authenticating messages, as well as inserting the appropriate capabilities for outgoing messages and validating messages on the receiving end. It is also responsible for enforcing delegation and revocation mechanisms. Details pertaining to these additional access control mechanisms can be found in more detailed work about the security infrastructure [48]. For domain-based access control (DBAC) (see Fig. 6), we assume the network is partitioned apriori, encapsulating nodes of various security levels (in this work we assume four levels, ranging from no security (L0) to high security (L3)). These domain partitions represent certain equivalence classes and allow one to control the granularity of the security of the domain. This ’security level’ (SL) of a domain is represented as a tag associated with a given domain and with each actor in the system. Base-level actors created in initial domains are assigned a SL which determines the degree of secrecy they can have (with respect to the state that possess) when moving between domains. When studying interactions between various services and policies in a distributed system, objects moving between different domains with varying security requirements should preserve a set of properties for availability, secrecy, integrity and semantic consistency to be maintained [48]. These properties should also maintain consistency within domains in regard to auxiliary object services interacting with or as part of the access control framework, such as object creation, garbage collection and forwarding.
17
3.4
Compositionality via the QoS Brokerage Service
In order to achieve the desired level of QoS in terms of application requirements and middleware service specifications, the base level component of the meta-architecture implements the functionality of the distributed session and deals with (a) data, which includes objects of varying media, types, e.g., video and audio files and (b) requests to access this data via sessions. The meta-level component deals with the coordination of multiple requests and sharing of existing resources among multiple requests. To provide coordination at the highest level and perform admission control for new incoming sessions, we created a meta-level entity called the QoS broker. The organization of the meta-level services in CompOSE|Q is illustrated in Fig. 7; the services have been mapped to a specific distributed server architecture [45, 38].
Fig. 7: A sample QoS brokerage for soft real-time services.
The two main functions of the QoS broker are (a) data management and (b) request management. The data management component decides the placement of data in the distributed system, i.e. it decides when and where to create additional replicas of data. It also determines when additional replicas of data actors are no longer needed and can be garbage collected/dereplicated. We implement adaptive admission control mechanisms [45] in the requestscheduling module that assigns requests to servers and ensures cost-effective utilization of resources. The message-scheduling module ensures QoS constraint satisfaction of requests that have already been initiated. The data and request management functions in turn require common object services such as clock synchronization, replication, de-replication and migration. For instance, (a) Replication: to replicate data and request actors using adaptive and 18
predictive techniques for selecting where, when and how fast replication should proceed; (b) De-replication: to dereplicate/garbage-collect data or request actors and optimize utilization of distributed storage based on current system load and expected future demands for the object; (c) Migration: to migrate data or requests for load balancing, availability and locality. The interaction of migration with timing based QoS constraints is an interesting issue since it can introduce playback jitter in MM applications caused by explicit teardown and re-establishment of network connections. In order to ensure non-interference among the common object services used to provide QoS, the specific mechanisms implemented for placement and scheduling must be designed not to conflict with each other. Currently, placement and dereplication operate on the basis of a (conservative) snapshot of the current resource allocation and use. The placement and dereplication services do not consider the exact times at which requests arrive; in contrast, an adaptive request scheduling process makes decisions based on the exact arrival times of requests. However, without appropriate constraints on the usage of these services, inconsistencies can arise due to their interaction. The broker coordinates the service interaction by constraining the behavior of the placement and scheduling services. For instance, the dereplication service does not dereplicate a replica that the request scheduling process is making an assignment to. Furthermore a replica assigned to an active request should not be physically dereplicated. The broker also ensures that the dereplication and placement metalevel services do not cancel one another out. While the interaction between dereplication and placement is not a functional correctness issue, it has to do with cost-effective performance of the overall system. Formal reasoning of QoS properties within a meta-architectural model, in the presence of other activities has been discussed in [38]. Another instance of providing safe composition between middleware services is that of secure video streaming. Here, the extra overhead of the security service could possibly introduce additional latency that will cause violations of real-time constraints. The brokerage must detect such violations caused by the composition of security and real-time video delivery services. Furthermore, it will exploit flexibility in video transmission quality (i.e. bitrate) to ensure adequate levels of security while maintaining jitter-free transmission of streaming video. The broker picks an optimal combination of transmission bitrate and security parameters (i.e. en19
cryption key-length, protocol) and the schedulability of the selected composition is verified by an admission control process at the RT module. Additionally, the broker may further exploit characteristics of the transmission path to further balance the quality-security tradeoffs. For example, nodes along the stream supporting flexible transcoding policies allow the bitrate to adapted, depending on the real-time requirements enforced by the request management module. The security framework in turn, allows the video stream (more specifically the packets) to be transcoded at intermediate points without having to decrypt them (which is handled by the security module at each node). This in effect, preserves end-to-end security, but is not a necessity if the intermediate nodes can be trusted (this in turn can be determined based on the domain-partitioning of the network and respective security levels associated with them).
4
Implementation & Performance of CompOSE|Q
The CompOSE|Q middleware consists of a set of runtime kernels that run on the individual nodes of the distributed system and distributed middleware components that provide the required distributed services. The node runtime kernel is a substrate on which actors execute [17]. The principal component of the node runtime kernel is a meta-level node manager actor. The node manager interfaces with a set of runtime components that implement basic services such as actor scheduling, message communication and actor creation. The distributed middleware layer of CompOSE|Q contains metalevel services such as remote creation, distributed snapshots, directory services, migration and a QoS brokerage. The distributed middleware components can execute on a single controller node or may themselves be distributed across the nodes in the system, e.g. the QoS broker operates on a single node, whereas the migration meta-actors are distributed across nodes.
4.1
Implementation of the Node Run Time Architecture
In implementing the run-time, we suitably ’constrain’ the Java programming language to achieve Actor semantics; this methodology is also taken in systems such as the Actor Foundry [2]. In order to assist the three core services in achieving their task easily and efficiently, the run-time system consists of:
20
The NodeManager: Each node running CompOSE|Q has one NodeManager to manage actors on that node, as well as to start-up and shutdown various other modules of the run-time system. When a new actor is created it registers itself with the NodeManager and is identified by an Actor ID (aid). The NodeManager enters the new actor into a local-table which helps keep track of the actor for activities such as node checkpointing and node shutdown and notifies the MessageQManager entity, which allocates a message queue that serves as the “in” Queue for the actor. The MessageQManager is responsible for managing mail queues for all actors on a node. To start CompOSE|Q on a node, the NodeManager has to be started first. It in turn, initiates the various other modules and communication components. The NodeInfoManager: The NodeInfoManager is a repository of information as well as an interface to the main directory service in the distributed architecture. Currently, the NodeInfoManager implements basic functionality to; (1) Register an actor with the directory service so that it is accessible to all other nodes; (2) Search for a particular actor to find out which node that actor is currently on and; (3) Search for an actor object given the class name (a rudimentary naming service). The NodeInfoManager has a local-table, which contains references to all local actors (updated and maintained by the NodeManager) and a remote-actor cache that contains information about recently accessed remote actors. The Communication Subsystem: The communication transport layer and the RCF module (above the transport layer), together compose the node communication subsystem. The message transport layer provides a framework for sending the outgoing messages to the appropriate node (routing) and resolving incoming messages to their appropriate actor queues (messageresolution). The RCF module is responsible for the implementation of communication services (and their composition). Separation of these layers allows for independent customization of protocol implementations and the messaging runtime. This facilitates correct composition of protocols without interfering with the runtime communication semantics. The communication transport layer consists of the following components: a Router, a Postman and a RemoteMessageReceiver (Fig. 8). The transport layer maintains two message queues on a node for incoming and outgoing messages (on that node) called SendPot and ReceivePot respectively. The Router consults the NodeInfoManager to obtain the current location (node)
21
Fig. 8: Schematic of the software design for the CompOSE|Q runtime environment.
of the target actor. If the location of the target actor is local, the Router puts the message directly into the node’s ReceivePot. Otherwise, it sends the message to the remote node. The RemoteMessageReceiver (RMR) on the target node extracts the incoming message and puts it into the node’s ReceivePot. The Postman then picks up the message and adds it to the target actor’s “in” Queue. The communication manager is instantiated in each node during system startup. When an actor is created and protocol composition services are not desired or required, its messenger is not created and the actor sends and receives raw messages using the transport layer. When flexible communication is required or desired, an independent messenger is created for every base level actor and the entire RCF functionality is invoked. The overhead of the RCF module is minimized in case of communications with no protocols attached. In this scenario, we tunnel raw messages through the actor’s messenger directly to the transport layer. Performance Results for the Run Time: Performance metrics of interest include timing for basic operations, and communication overheads. All execution times presented here are average execution times over 100 iterations, without JVM induced overheads. The execution times of the various basic operations are summarized in Tables 1-4 (see Fig. 9). A locality check is used to determine whether an actor is local or remote and executes in 31 µs. In order to measure end-to-end message overhead, we measured message transmission 22
Fig. 9: Performance results of the run-time.
and reception of the underlying transport layer, which serve as a reference for the RCF overhead, then we measured message tunnelling. Finally we measure transmission of processed messages. A local send/dispatch operation executes in 181 µs. The timings of the remote message send can vary dramatically depending on whether the location of the target actor exists on the remote actor cache on a node (cache hit) or not (cache miss). With a cache hit, a remote send operation executes in 254 µs, whereas a cache miss causes a remote send to take 42,630 µs to execute. The performance impact of caching (at the socket cache and remote actor cache) can be seen in the effective roundtrip latencies for an initial message (approx. 3 secs) and subsequent messages (50ms). The RCF overhead varies between 28.67–237 µs in message reception and between 2091296 in message transmission, depending on the number of protocols used (a protocol consumes approx 45 µs) and whether or not tunneling is applied.
4.2
Implementation of Meta-Level Services
Here we describe the implementation of a few meta-level services being developed for the CompOSE|Q framework. For further details on the implementation of metalevel services see [42]. Our initial implementation of the directory service (DS) uses the Lightweight Directory Access Protocol (LDAP) [49], which is easily adaptable to many commercial directory services such as NDS (Novell Directory Service). Recent work on efficient implementation of LDAP query processing includes directory caching and flexible management of lists [19]. We utilize the OpenLDAP Group’s slapd LDAP server [30] for our implementation. We used the Berkeley Sleepy Cat Database as the backend for slapd. An abstraction layer was implemented that
23
presented only the necessary functionality via the Netscape Java LDAP API. Following preliminary testing of the DS, tuning was done to ensure faster performance for both access and updates. Caching was also used, both for the entries and indexing. These parameters were tuned to the system, and a variety of tests done to determine the optimal settings. Directory access overhead is substantially high in comparison to the native execution time of basic operations. We are currently working on removing directory access out of the critical datapath for frequently used operations. We are also studying scalability issues in directory services. Our initial attempts at performance optimization have yielded significant improvements over raw access and update overheads. Adding an actor takes 27,463µs, which is an 80% improvement over raw access times. Adding an actor attribute and searching for an attribute take 5,875 & 2814µs respectively (90% improvement over raw overheads). The current implementation of remote creation only allows for the creation of a single actor α on a remote node ν with an initial state σ. A remote creation request takes the form RC (α, σ, ν, β). This call creates and starts an actor, α with the specified state, σ on the remote node, ν. The call is made to the RCM, which then uses the NodeManager (NM) to implement remote creation. Remote Creation proceeds in two phases. In the first phase the two NMs (caller and callee) go through a handshake protocol to negotiate loading of classes and execute checks for security and resource availability. If the handshake is successful, then the caller NM transfers the actor-state, σ, to the callee node, ν, which then initializes a newly created instance of the actor-class with that state. As described in the architecture of CompOSE|Q the migration service is built using the remote creation core service. Migration behavior is specified by assigning to each node a migration Meta actor (MM) that handles migration requests for actors on that node. A remote creation service accessed via a remote creation request RC is used to install the migrating actor’s state on the remote node. The remote creation request also includes a message to be sent to the original node containing the address of the newly created actor. To avoid confusion with other messages to the migrating actor α, a temporary surrogate actor αs is created to receive this message. Performance results for the migration and remote creation services are depicted in Table 1 (see Fig. 10). The results are an average over 10 runs. The main overhead in all these 24
operations is message passing between nodes. A round-trip between nodes takes an average of 41 ms. A Remote Creation call involves at least one round-trip message (phase 1). Thus, excluding the messaging overhead, remote creation takes only about 7 ms, which is just 1 ms more than an local actor create. In migration there are more messages being passed around, so the timings are expectedly higher (82 ms). The Migration and finalization phase includes the time for a remote creation (48 ms) plus the waiting of the surrogate actor for a message from the migrated actor.
Fig. 10: Performance results for meta-services.
Scheduling: The peripheral meta scheduler module runs on each individual node in the distributed system and it schedules locally originating tasks along with the tasks assigned to it by the focal scheduler. This scheduler performs the conjunctive message and task scheduling based on the system wide meta-rules established by the focal scheduler. Each peripheral node is aware of the different message and task priority levels and can therefore generate a unique priority ordering based on the independent task and message priorities. Any global changes made by the focal scheduler are reflected in the directory service and the peripheral schedulers periodically update their local information from the directory service. Therefore, there might be some delay before the global policies are reflected in all the distributed nodes. Fig. 4 shows the changes required to the node runtime to achieve conjunctive scheduling. The conjunctive scheduling rules are used to determine the final task priorities. At this point, it is important to note that the PMS performs scheduling based on the tasks running on its node and the messages inbound to them. In order to ascertain the individual message priorities, the scheduler needs to peek into each message. Within CompOSE|Q the Postman routes all messages to their destination queues and is the only entity that touches each message. We therefore introduce a small scheduling agent into the Postman that reads the individual message priority. The individual actor message queues are also modified to support priority messages. i.e. 25
messages can be inserted into the message queues based on their priorities (no longer a FIFO queue). Real time actors are not affected by the conjunctive priority assignments and always take precedence over the non realtime actors. The sub scheduler has a multi-threaded design and is composed of the Broker and the Dispatcher. Performance results for the RT scheduling service are depicted in Table 2 (see Fig. 10). Dispatch latency is the length of time to switch from the end of one time-slot to the beginning of the next time-slot (scheduling overhead that should be kept to a minimum). The dispatch table calculation time is the time taken by Broker to calculate the dispatch table. Finally, the scheduler response time is the time elapsed between the application base actor contacting the broker and getting the reply. Access Control: The prototype security modules have also been implemented in Java to facilitate portability, flexibility through introspection and type-safety. Additionally, they allow first-class capability objects to be implemented in an object-oriented manner and allows for delegation of these capabilities through sub-classing. Capabilities are defined as the following triple: ActorID x Access Rights x CapInf and represented as first class objects. ActorID uniquely identifies an actor within the system (address), the Access Rights define valid operations for the object. Finally, CapInf allows additional information to be embedded into the capability (e.g. expiration, where a timestamp is utilized to indicate a period of validity for the capability). An additional field allows the capability object to be signed cryptographically. Each (local) MSA implements a caching mechanism on each node, that allows a table of rights to be maintained in order to minimize directory accesses. In addition, for remote creation and migration, a secure class loading mechanism was built on top of the existing remote class loader available in CompOSE|Q [43]. The MSA encapsulates the behavior of the security service on each node, and is responsible for creation, validation & management of capabilities. At present, the behavior of the MSA is pre-defined but we are working on adding the ability to define customized user-policies on each node (MSA). This is not a trivial matter, as interdomain coordination of these policies must be dealt with (an ability being implemented within the global MSA). At present the MSA implements the following functionality: (1) Registration with the local Node Manager.; (2) Installation of access rights for existing base-actors.; (3) Message tagging.; (4) Access rights checking (enforcement). 26
During the installation process (2), the MSA queries the Node Manager and (via the name service) obtains a list of actors present on the given node. Based on the security policy at the MSA it installs access rights for the actors (default rights if there are none specified), updates the directory service with the newly defined access control information and caches the result locally. The MSA also handles installation of rights for newly created actors. Object creation signals the MSA on the node of creation and the new capabilities are dynamically added to the rights table. The enforcement (4) is carried out at the CompOSE|Q transport layer [43], which currently uses TCP sockets for messaging and caches open connections (for reuse). The CompOSE|Q transport layer depicted in Fig. 8 provides an illustration of the various components involved in this filtering process (namely the SendPot and ReceivePot where the message interception occurs). Performance results for the access control service are depicted in Tables 1-2 (see Fig. 11). In Table 1, the results are divided into the MSA bootstrapping and the MSA operational overhead. To determine the startup overhead, we measured the execution time of the MSA initialization & registration processes. InstallRights represents the time taken
1
for the MSA to obtain a
list of actors (local send & receive RTT) on its node and setup access rights for them (local store). All times depicted here are for local operations (assuming that the required information is available in the local cache) and do not include interactions with the directory service. Table 2 illustrates the overheads of the various MSA operations. On message send, we measure the raw processing overhead (does not include cryptographic overhead of signing the capability) necessary to tag outgoing messages (i.e. from when the message is intercepted by the MSA to when it is returned to the regular messaging system) 2 .
Fig. 11: Performance results for the access control service. 1
measured for 10 actors Note that these measurements do not include the overhead for message interception and re-insertion into the communication subsystem. 2
27
5
Related Work
Commercially available object-based middleware infrastructures including CORBA and DCOM represent a step toward compositional software architectures but do not deal with interactions of multiple object services executing at the same time, and the implication of composing object services. The Electra framework [26] extends CORBA to provide support for fault tolerance using group-communication facilities; real-time extensions to CORBA [34] to support timingbased QoS requirements have been proposed. Systems such as dynamicTAO [20] explore ways to make the various components of an ORB dynamically configurable as well as componentizing them to achieve minimal footprint for small applications [50]. Other systems such as Infospheres [7] and Globe [36] use a distributed object model to construct large scale distributed systems. Globus, a metacomputing framework defines a QoS component called Qualis [22] where low level QoS mechanisms can be integrated and tested. The QuO project from BBN [51] deals with the specification, monitoring and control of application level QoS. QualMan [28] is a QoS aware resource management platform that provides negotiation, admission and reservation capabilities for sharing end-system resources. To our knowledge, none of the above systems provides a formal semantics for the composition of middleware services and protocols. The distinguishing feature of the TLAM architecture on which the CompOSE|Q system is based is that it is based on formal semantics that supports safe and correct composability of the services being implemented. This in turn, provides a framework for the dynamic composition and safe adaptation of middleware services that can provide application level QoS guarantees. Traditional reflective languages and systems aim at providing a customizable execution of concurrent systems [32]. The Aspect Oriented Programming paradigm [18] makes it possible to express programs where design decisions (aspects) can be appropriately isolated permitting composition and re-use of code. Reflective systems being developed include Apertos [15], 2K [21] and Broadway [33]. Recent work integrates the OpenORB architecture [5] with the Aster system [14] to apply ideas from the software architecture community to the field of adaptive component-based middleware. The distinguishing feature of the CompOSE|Q architecture is that it is based on formal semantics that ensures safe and correct composability of the services being implemented. In other reflective models for distributed object computation [31, 5], an
28
object is represented by multiple models allowing behavior to be described at different levels of abstraction and from different points of view. In the TLAM, each meta actor can examine and modify the behavior of a group of base level actors – namely those located on the same node. In former approaches to dynamic installation of communication services, e.g. Broadway [33], Actor Foundry [2], multiple protocols may be applied to a single component by stacking metalevel objects, which implement each protocol. The pluggable protocol framework [3] addresses the lack of support for multiple inter-ORB protocols and deals with integration and use of multiple ORB messaging and transport protocols, not with the composition of the protocols itself. Theoretical and formal approaches [27, 4] study modular and adaptable solutions that deal with the problems posed by composition of distributed communication services. Many such approaches assume that communication is point-to-point. Specification and formal characterization of group communication services has been developed in the context of I/O Automata [24] and in recent years, several group communication systems have been developed to provide additional flexibility of communication services by allowing for dynamic composition of communication protocols [37, 13]. The RCF approach presented in this paper enforces formal restrictions on the structure of basic communication primitives to facilitate dynamic installation and composition of communication protocols. CompOSE|Q also facilitates the customization and composition of protocols and services using a single unified framework.
6
Conclusions
The need for formal semantics and correctness reasoning has generally been underestimated by system developers. With the growth in distributed computing environments, middleware frameworks are starting to employ techniques such as computational reflection to manage resources and services under constantly changing conditions. However, while reflection is a very powerful tool for providing customizability, it can violate desired application semantics and cause unexpected behavior when used in an unconstrained manner. Interaction constraints that ensure application and system level correctness should typically be transparent to the application designer and handled by the middleware framework. The two-level architecture presented in this paper provides a framework that allows structured dynamic customization of middleware services in a correctness-preserving manner. The 29
two-level approach naturally extends to multiple levels, with each level manipulating the level below while being protected from manipulation by lower levels. In practice, however, expressing a computation in terms of multiple meta-levels becomes unwieldy. A purely reflective architecture provides an unbounded number of meta-levels with a single basic mechanism. The formal verification of interaction semantics between the different layers in the reflective hierarchy can be quite complex and requires further investigation. We are currently implementing a number of extensions to the CompOSE|Q runtime. For instance, we are actively working on extending CompOSE|Q to support more distributed services for scheduling, security, mobility and fault-tolerance. We are developing directory-enabled mobility management techniques that can support QoS requirements of applications executing under poor conditions of network connectivity. Power-cognizant middleware management mechanisms are being designed to maximize device lifetimes while guaranteeing user QoS. Efficient support of security in ubiquitous environments will entail the integration of application, network, middleware and OS level security strategies that can be tailored to the needs of the user and application. We are also exploring techniques for designing and managing intelligent network infras˙ tructures using the basic services in CompOSE|QThese include services to support dynamic network customizations that deal with routing and network management (information collection) that adapt to application and system conditions. These services implement decisions about the degree of network awareness that applications and middleware must possess to ensure performance under varying network conditions. Further work is required to provide a generalized model that captures the architectural resources required in the server and network to support a session connection. We are further refining the concept of QoS Synchronizers [38] and developing techniques for their specification, use and implementation. In general, the dynamic nature of applications under varying network conditions and request traffic imply that middleware mechanisms must be dynamic and customizable. We believe that composable and safe middleware frameworks that implement cleanly defined meta-architectures enable customization of applications, protocols and system services; this will provide a foundation for the evolution of large scale distributed computing.
30
References [1] G. Agha: Actors: A Model of Concurrent Computation in Distributed Systems. MIT Press, 1986. [2] Mark Astley & Gul Agha: Customization and Composition of Distributed Objects: Middleware Abstractions for Policy Management. Intl. Symposium on the Foundation of Software Engineering, 1998. [3] A.B. Arulanthu, Carlos O’Ryan, Doug Schmidt, Michael Kircher & Jeff Parsons: The Design and Performance of a Scalable ORB Architecture for CORBA Asynchronous Messaging. Proc. of Middleware, New York, 2000. [4] Lynne Blair, & Gordon Blair: The Impact of Aspect-Oriented Programming on Formal Methods. Proceedings of ECOOP Aspect-Oriented Programming Workshop, Brussels, 1998. [5] Gordon Blair, Geoff Coulson, Phillipe Robin & Michael Papathomas: An Architecture for Next Generation Middleware. In Proc. of Middleware’98, Lake District, England, November 1998. [6] Chandy, K. M. & Lamport, L: Distributed Snapshots: determining global states of a distributed system ACM Transactions on Computing Systems, 1985. [7] K. Chandy, et al.: A Worldwide Distributed System using Java and the Internet Symposium on High Performance Computing (HPDC-5), Syracuse, New York, Aug. 1996. [8] Benoit Garbinato & Rachid Guerraoui: Using the Strategy Design Pattern to Compose Reliable Distributed Protocols. Usenix Conference on Object-Oriented Technologies and Systems, 1997 [9] Benoit Garbinato & Rachid Guerraoui: Flexible Protocol Composition in BAST IEEE International Conference on Distributed Computing Systems, 1998 [10] Erich Gamma, Richard Helm, Ralp Johnson & John Vlissides: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1994. [11] Sebastian Gutierrez-Nolasco & Nalini Venkatasubramanian: Design Patterns for Safe Reflective Middleware. OOPSLA, Workshop Towards Patterns and Pattern Languages for Object-Oriented Distributed Real-Time and Embedded Systems, 2001. [12] Helary, J.: Observing global states of asynchronous distributed applications Proceedings of the Workshop on Distributed Algorithms, Springer-Verlag, 1989. [13] Mark Garland Hayden: The Ensemble System PhD Thesis, Department of Computer Science, Cornell University, 1998. [14] V. Issarny & C. Bidan: Aster: A Framework for Sound Customization of Distributed Runtime Systems Proc. of ICDCS’96.
31
[15] J. Ichiro Itoh, R. Lea, & Y. Yokote: Using meta-objects to support optimization in the Apertos operating system In USENIX COOTS (Conference an Object-Oriented Technology, June 1995). [16] James B.D. Joshi, Walid G. Aref, Arif Ghafoor, & Eugene H. Spafford: Security Models for Web-Based Applications. Communications of the ACM, Vol 44, No. 2, Feb 2001 [17] Wooyoung Kim: Thal: An Actor System for Efficient and Scalable Concurrent Computing PhD thesis, UIUC, 1997. [18] G. Kiezales, et al.: Aspect-Oriented Programming In Proceedings of ECOOP’97, June 1997. [19] Olga Kapitskaia, Raymond T. Ng & Divesh Srivastava: Evolution and Revolutions in LDAP Directory Caches Proc. of the EDBT 2000. [20] Fabio Kon, et al.: Monitoring, Security, and Dynamic Configuration with the dynamicTAO Reflective ORB IFIP/ACM Middleware’2000. New York. April 3-7, 2000. [21] F.Kon, et al.: 2K: A Reflective, Component-Based Operating System for Rapidly Changing Environments. In Proceedings of ECOOP’98 Workshop on Reflective Object-Oriented Programming and Systems, Brussels, Belgium, July 1998. [22] C. Lee et al.: The Quality of Service Component for the Globus Metacomputing System Proceedings of IWQoS ’98. [23] R.G. Lavender & Douglas C. Schmidt: Active Object: an Object Behavioral Pattern for Concurrent Programming. Pattern Languages of Program Design, 1996. [24] Nancy Lynch & M R Tuttle.: An Introduction to Input/Output Automata CWI Quarterly, 2(3):219-246, 1989. [25] Leslie Lamport: Time and clocks and the ordering of events in a distributed system Communications of the ACM,21(7):558-565,1978. [26] S. Maffeis & D. Schmidt: Constructing reliable distributed communication systems with CORBA IEEE Communications, 14(2), February 1997. [27] Jose Meseguer, Carolyn Talcott & Denker G: Rewriting Semantics of Meta-Objects & Composable Distributed Services 1999. [28] K.Nahrstedt, H.-H. Chu, & S.Narayan.: QoS aware resource management for distributed multimedia applications Journal on High-Speed Networking, Special Issue on Multimedia Networking, 1998 [29] Priya Narasimhan, Luise E. Moser & P.M. Melliar-Smith.: Using Interceptors to Enhance CORBA IEEE Computer, 1999. [30] OpenLDAP: http://www.openldap.org/
32
[31] Okamura, H. Ishikawa, Y. & Tokoro, M: A Distributed Programming System with MultiModel Reflection Framework ACM SIGPLAN, 1992. [32] B.C. Smith: Reflection and Semantics in a Procedural Language. PhD thesis, Massachusetts Institute of Technology, Jan 1982. [33] D.Sturman: Modular Specification of Interaction Policies in Distributed Computing PhD thesis, University of Illinois at Urbana-Champaign, May 1996, TR UIUCDCS-R-96-1950 [34] D.C. Schmidt, D.Levine, & S.Mungee: The Design of the TAO Real-time Object Request Broker. Computer Comm. Special Issue Building Quality of Service into Distributed System, 1997. [35] Ashish Singhai, Aamod Sane & Roy Campbell: Reflective ORBs: supporting robust, timecritical distribution ECOOP 1997. [36] M. van Steen, A. Tanenbaum, L.Kuz, & H.Sip: A Scalable Middleware Solution for Advanced Wide-area Web Services In Proc. Middleware ’98, The Lake District, UK, 1998. [37] Robbert van Renesse, Kenneth Birman & Silvano Maffeis: Horus: A Flexible Group Communication System Communication of the ACM, 39(4):76-83, 1996. [38] N. Venkatasubramanian: An Adaptive Resource Management Architecture for Global Distributed Computing. PhD thesis, UIUC, 1998. [39] Carlos Varela & Gul Agha: Linguistic Support for Actors, First-Class Token-Passing Continuations and Join Continuations. Midwest Society for Programming Languages and Systems Workshop, 1999 [40] Carlos Varela & Gul Agha: Programming Dynamically Reconfigurable Open Systems with SALSA. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), Intriguing Technology Track, 2001. [41] N. Venkatasubramanian, G. Agha, & C.L. Talcott: Scalable distributed garbage collection for systems of active objects. In International Workshop on Memory Management, IWMM92. [42] N. Venkatasubramanian et al.: Design & Implementation of CompOSE|Q. Technical Report, University of California, Irvine, 2000. [43] N. Venkatasubramanian, M. Deshpande, S. Mohapatra, S. Gutierrez-Nolasco & J Wickramasuriya: Design and Implementation of a Composable Reflective Middleware Framework. IEEE International Conference on Distributed Computer Systems (ICDCS-21), April 2001. [44] Robbert van Renesse, Kenneth Birman & Silvano Maffeis: Horus: A Flexible Group Communication System. Communications of the ACM, 39(4):76-83, 1996 [45] N. Venkatasubramanian & S. Ramanathan: Effective load management for scalable video servers. In Proceedings of ICDCS97, May 1997.
33
[46] N. Venkatasubramanian & C.L. Talcott: Reasoning about Meta Level Activities In Open Distributed Systems. In 14th ACM Symposium on Principles of Distributed Computing, 1995. [47] Victor F Wolfe, Lisa C DiPippo, Roman Ginis, Michael Squadrit, Steven Wohlever, Igor Zykh & Russell Johnston: Real-Time CORBA. Department of Computer Science & Statistics, University of Rhode Island, TR97-25b, 1997 [48] J. Wickramasuriya & N. Venkatasubramanian: A Middleware Approach to Access Control for Mobile, Concurrent Objects. DOA 2002. [49] M Wahl, T Howes & S. Kille: LDAP (v3). IETF RFC 2251, December 1997. [50] Manuel Roman et al.: LegORB and Ubiquitous CORBA. Proceedings of the IFIP/ACM Workshop on Reflective Middleware (RM2000), April 2000. [51] J. Zinky, D.Bakken & R. Schantz: Architectural Support for Quality of Service for CORBA Objects. In Theory and Practice of Object Systems, pages 41-49, 1997.
34
