Middleware for Large Distributed Systems and Organizations George Dimitoglou1 Philippe Moore2 and Shmuel Rotenstreich3 Dept. of Computer Science The George Washington University Washington, DC Abstract Modern computational environments are complex, dynamic and highly distributed. These environments include multiple independent or grouped objects that communicate, collaborate and perform various activities. We describe Middleware++, a unifying system and computational construct that makes the use and operation of these large, fully distributed environments feasible and manageable. In this paper we present the basic Middleware++ mechanisms that provide the computational building blocks for constructing and managing dynamic relationships in rapidly changing, decentralized environments. Keywords: middleware, distributed systems, dynamic organizations 1 Introduction In this paper we present Middleware++ a new computational model for organizing and managing interactions in distributed environments comprised of multiple participants (i.e. objects, groups, resources, organizations) that rely heavily on information and communication infrastructures to arrange and coordinate various activities. These activities may cross over organizational and functional boundaries. Middleware++ is a middleware that provides an infrastructure to ensure that independent groups or organizations collaborate, achieve authority and maintain accountability to achieve common objectives. It serves as a shared collaboration medium mediating reliable and transparent access to remote assets, irrespective of their (network) location or underlying platform. It brings together geographically dispersed, distributed entities toward accomplishing a specific task. Conceptually, Middleware++ environments provide a shared context and the transactional boundaries for ongoing activities resembling human organizations. These organizations are dynamically formed to support particular activities or missions. Today, large corporate and military organizations are typical examples of such environments. In the first section we describe the system architecture and the basic components of Middleware++. The second section provides an overview of the tasking model and some of its features such as task decomposition and negotiation. The third section introduces the notion of teams as one of the many supported organizational structures. The final section covers management facilities and examines the Logging mechanisms as an example of these facilities. 2 System Architecture and Organization We assume the existence of an underlying physical network infrastructure. The services provided by this infrastructure may range from simple routing services (like the Internet) to complex network constructs consisting of both physical and logical interconnected components (Fig. 1). 2.1 Objects Middleware++ objects are instantiated entities with a variety of functionality and features. Objects provide numerous services ranging from simple message forwarding to elaborate negotiating, recruiting and team forming functionality. This is accomplished by implementing with facilities to enable communication, message handling and task management along with a versatile Application Programming Interface (API) to allow further extension of their functionality. Objects typically consist of three components: communication, message-handling and task management. The communication component facilitates communication between objects and their environment. The message-handling component behaves as a parser and processor of 1
[email protected] [email protected] 3
[email protected] 2
Copyright held by the author
536
exchanged messages. In the case of incoming communication, messages are parsed and then sent to the appropriate object component for processing. In the case of outgoing communication, the message-handling component acts as a packager that assembles messages in a Simple Object Access Protocol (SOAP) envelope for transmission. Finally, the task management component handles any tasking and monitoring requests that the object may issue or be issued to perform. While Middleware++ objects can act independently, their activities are more interesting and complex when they form loosely coupled communities. The type of community depends on the function its objects perform. We identify three generic organizations of objects. 2.1.1 Unstructured objects Some environments consist of unstructured collections of objects. Objects in these collections do not belong to pre-defined hierarchies or control structures. Their lack of belonging to a structure does not preclude them from interacting with each other. In fact, such interactions are the norm and such unstructured organizations are common in many distributed environments. 2.1.2 Hierarchies of objects Hierarchies imply that some objects have a “parenting” role over other objects lower in the hierarchy. These roles may range from task delegation to more complex authority and accountability relationships. Middleware++ objects have built-in knowledge of hierarchical object structures. Objects are aware of their parents (higher level objects) and their children (lower level objects). The notion of hierarchy becomes very helpful in a variety of distributed settings. In a system controlling sensors, for example, a group of sensors may report to an object performing data analysis on sensor output. When multiple, often spatially dispersed, sensors are involved a single processing object may not have sufficient perspective in evaluating the collected data. Hence, a higher-level object may receive the data from the geographically, thematically, electronically restricted objects to process and form a more comprehensive picture. 2.1.3 Teams Teams are formed when objects (members) share resources and work together in a loosely or tightly coupled way to perform a single or a set of collaborative tasks. Middleware++ offers a full spectrum of coupling and cooperation options. A team may be advertised for discovery purposes as a single service, using individual member resources to complete tasks. However, team members continue to maintain their identity, properties and capability of performing tasks outside team boundaries (see sec. 6). In Middleware++ the types of organizations are not mutually exclusive. An object may belong and collaborate with members of multiple teams, while communicating with third-party objects and teams using peer-to-peer (P2P) mechanisms. At the same time, the object may be performing tasks as an unstructured object or as a part of a Middleware++ hierarchy. Essentially, one sort of organization does not preclude non-empty intersection with other types of organization. 3 Communication We use the SOAP [Simp] and Extensible Markup Language (XML) [Haro] messages for communication. SOAP makes communication simple and platform independent. XML makes exchange of different components simple to define and use. Each organizational type communicates differently. Unstructured objects may use direct P2P communication or multicasting. Objects within hierarchies may communicate bi-directionally following the tree-like hierarchy (downwards and upwards) or communicate as unstructured objects.
537
Fig. 1: System Architecture
4 Resource Discovery & Service Directories Objects, resources and services are indistinguishable in Middleware++ and all must register with their local service directory. Service directories facilitate resource discovery within the distributed environment. If a local directory does not exist, or has failed, objects search for other directories to register. The service directory supports three operations: (a) Registration. Objects register with a service directory and objects are then advertised so other objects may locate and communicate with them. (b) Status Refresh. Upon registration, objects submit their expected lifetime. After their lifetime is expired, the directory service discards the object’s entry as no longer valid. Objects and services fully control their availability. They may renew their lifetime by periodically sending Status Refresh messages. (c) Deletion. Objects may remove their entries from the directory service. The contents of the service directories include information about objects and services. Besides basic addressing (i.e. IP address in a TCP/IP network), they also provide specific and generic names. Specific names are unique and identify a particular object or service. Generic names identify one or more attributes of an object or a service. The distinction between specific and generic names is important. Discovery using a generic name results in a list of all objects with that name. Consequently, a service request for “plumber” results in all plumbers in the local directory. Discovery using a specific name results in the addressing information of the matching object. For example, a request for “Bill’s Plumbing” will return the service and access information of the particular service. This technique for identifying objects in a distributed environment is well tested, extensively used [Adji], and provides flexibility during the resource discovery process. Furthermore, multiple service directories may be replicated and loosely interconnected. Each directory maintains registration information of at least one additional directory enabling the transmission of contents to assist in recovering from directory failures. 5
Task Management
Task Management is the component that deals with the management of the environment’s activities. It is a distributed service implemented as a set of APIs that each service must use. It deals with the sending, receiving and queuing of activity announcements, evaluating and responding to bids, awarding tasks and notifying objects about task status and completion. We have developed two distinct activity types handled by task management: tasks and commands. Tasks are the units of work within Middleware++ and they are computations that require services. Task examples are requests to use a resource or orders to perform an activity (i.e. requesting an area scan from a satellite). Commands are computations that request organizational changes. Command examples are: spawn or move a service, create a team and recruit objects. 5.1 Handling Tasks The task manager handles both tasks and commands. In Middleware++ all message types (i.e. task, command, bid, status update, completion report) are represented using XML.
538
Fig. 2: Example of (a) task and (b) command XML messages.
In the XML task description (Fig. 2a) the task contains various information including the task recipient and other pertinent task parameters. Similarly, the XML command description (Fig. 2b) outlines the command parameters. A task may include several sub-tasks and can be sent to a specific object, service, team or group within the distributed environment. 5.2 Task Decomposition Once a task is sent to an object, the object attempts to decompose it to smaller tasks. This is accomplished by querying an XML file that describes how tasks may be decomposed to sub-tasks. If the decomposition fails, the object manages the received task as a single task. Otherwise, the object manages the set of generated sub-tasks. Tasks and sub-tasks may be inter-dependent. The execution order is enforced by dependencies contained in a tag within each task XML description. The syntax describing a dependency consists of an arbitrary unique number, followed by a decimal point, followed by at least one other number (Fig. 1). The numbers following the first decimal point are the tasks upon which this task is dependant on.
Fig. 3: Example and conceptual representation of task dependencies
The use of dependencies may be demonstrated using a fighter-jet pre-flight preparation scenario. Pre-flight preparation includes tasks such as fuel the plane, load ammunition, brief the pilot about the upcoming mission and run final pre-flight checks. Some of these tasks may be performed in parallel since they are not dependent on each other (i.e. pilot may be briefed while the plane is being fueled) while others must be performed in a certain sequence (i.e. performing final pre-flight checks only after all other preparation activities have been completed). Overall, task dependencies ensure the serial execution sequence of some tasks and the parallel execution of others. 5.3 Task Distribution and Negotiation After decomposition, sub-tasks are treated as discrete tasks. The object handling all of the subtasks plays the role of the Task Initiator (Fig. 4) and prepares time-sensitive task announcements for each sub-task. Then the Task Initiator sends each task announcement via a controlled broadcast to only the objects with generic names matching the task type to be processed. This activity initiates the first phase of a negotiation process.
539
Fig. 4: The five phases of task negotiation in Middleware++
The objects that receive the task announcements evaluate them and, if capable of completing the task, respond with bids. The bids contain task performance related data such as expected execution duration, required resources to complete the task and other information. The second phase is the bid evaluation phase when or after the Task Initiator collects the bids. Various algorithms may be employed to evaluate incoming bids. Middleware++ offers two algorithms with the flexibility to add others if necessary. One algorithm collects bids for a certain time period and then selects the bid with the best terms. Another algorithm evaluates incoming bids against specific criteria and the first bid matching the criteria is selected. If the task announcement expires and no bid is received, the task announcements may be re-sent to a wider community of objects. If this also fails, the task may be announced at a later time in expectation that bids may be generated by new objects, or by objects that have recuperated their resources [Rose, Smit]. The third and fourth phases of the negotiation process award and commit the task to the object with the winning bid, following a two-phase commit (2PC) protocol. Once the task is awarded, the object executes and eventually completes the task. At that time, a task completion message is sent to the task initiator. Failure to send a completion message, or actual task completion failure triggers the negotiation mechanism to take remedial action. 5.4 Task Failures After the tasks have been announced, negotiated and awarded, they are executed. Middleware++ is designed for deployment in highly distributed, heterogeneous and perhaps volatile environments. Such settings may experience various failures during task execution, ranging from loss of communication to complete service failures. On such occasions, task execution is considered aborted or failed. In either case, a mechanism exists that acknowledges these conditions and attempts to save any of the work already performed. We employ a commitment and de-commitment mechanism to properly handle aborted and failed tasks. We assume that object commitment is the agreement to execute a task. De-commitment occurs when the Task Initiator receives a “failure to complete” message from the working object. It may also occur if the expected completion time expires and the Task Initiator is unable to reach the working object. These criteria reflect the Task Initiator perspective. From the working object’s perspective, decommitment occurs when execution of a current task is pre-empted (i.e. due to an order from a higher authority) for the execution of a different task. The working object then sends a message to the Task Initiator acknowledging its inability to complete the awarded task. Once the decommitment of a task is confirmed, the Task Initiator may abort the task entirely, or re-commit it to another object. 6 Team Formation & Recruitment New team member recruitment proceeds in three stages: • Invitation. Team leader invites specific objects or certain object types to join. Invitees may accept or reject the invitation. The exception occurs when the invitation is propagated as an order from a higher authority, in which case objects are forced to accept and join.
540
Joining. Joining a team may be done two ways. The first way is simple: upon acceptance by invited objects to join, the team leader confirms their new status as team members. The second way is more complex: requiring invitees to have appropriate authorization credentials and resources to join. If they fail to provide them, they may negotiate a deal with the team leader. • Data Propagation. New team members confirm they have joined the team and request team information, such as a team member list. Team membership however, is dynamic. Based on the changing team needs or newly assigned activities, a team may have insufficient resources to fulfill an activity. We support Resource-based Dynamic Recruitment, which allows the team to dynamically assess its needs and, if necessary, recruit new team members. This occurs after a team leader receives an activity and begins managing its delegation. If the leader discovers that not enough resources exist within the current team, the leader may recruit more resources. •
7 Auxiliary Facilities Middleware++ represents dynamic and complex organizations it is therefore essential that environments of this kind are, to some extent, manageable and monitored. For this purpose, we provide extensive Graphical User Interfaces (GUIs) and logging facilities to achieve real-time monitoring of objects and activities. As an example, we describe the Middleware++ logging mechanisms. 7.1 Logging Organizations are typically concerned with accountability. Logs are the means of revealing sequences of events that lead to the unfolding of accountability chains. Logging records transactions, activities and exceptions that may occur during system operation. When logs are examined they may also display causal relationships between events, and identify issues that may require attention. Middleware++ is implemented with an initial set of 50 types of log messages. However, there are no limitations in extending the number and type of messages. Existing messages provide logging capabilities ranging from simple activities (i.e. object creation, team join) to elaborate ones (i.e. negotiation, bidding, recruiting). The logging mechanism architecture may be centralized or distributed. With centralized logging, distributed objects log their events and activities to a central logging engine. This scheme provides a single point of monitoring and is implemented using a relational database or a Web server to maintain logs. With distributed logging, multiple logging engines are deployed throughout the environment with each logging engine treated as a service subject to discovery. In case of failure, an auto-discovery mechanism is initiated to locate an alternate logging engine. Distributed logging is somewhat fault-tolerant, fully scalable and avoids log traffic bottlenecks. In both the centralized and distributed schemes, Middleware++ also provides real-time and post-mortem (i.e. playback) log viewing functionality. 8 Conclusion In this paper we have described the architecture and the basic features of the Middleware++ infrastructure. First, we presented the basic system elements (objects, tasks) and mechanisms (communication, resource discovery). Then we described various organizational structures the elements may construct and introduced the task management functionality, which is essential in performing activities in a distributed environment. Along with the task management mechanisms of announcing, delegating and checking the status of tasks, we have described in detail the built-in task decomposition and negotiation mechanisms. Finally, we offered a close examination of teams as an object organization mechanism and logging as a sample of the Middleware++ management facilities. Overall, the Middleware++ infrastructure provides a very agile and flexible platform for the deployment of highly scalable and distributed environments.
541
Acknowledgements The authors are grateful of the contribution of the entire Middleware++ team for the development of the prototype. The work in this paper has been partly funded by the Dept. of Computer Science at The George Washington University and the Defense Advanced Projects Agency (DARPA). Bibliography [Adji] Adjie-Winoto, W., Schwartz, E., Balakrishnan, H., Jeremy Lilley. (1999) The Design and Implementation of an Intentional Naming System. Operating Systems Review, 34(5):186-201. [Dign]F. Dignum, B. Dunin-Keplicz, and R. Verbrugge. (2000) Agent theory for team formation by dialogue. In C. Castelfranchi and Y. Lesperance, eds., Proceedings Agent Theories, Architectures and Languages (ATAL 2000). [Haro] Elliotte Rusty Harold and W. Scott Means, XML in a Nutshell, 2nd Edition, O’Reilly. [Rose] Rosenchein, J.S., Zlotkin, G. (1994) Designing Conventions for Automated Negotiations. AI Magazine, p. 29-46. [Simp] Simple Object Access Protocol, 1.1, http://www.w3.org/TR/2000/NOTE-SOAP20000508. [Smit] Smith, Reid G. (1980) The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers 29(12):1104—1113.
542
