The Influence of Coordination on Program Structure Farhad Arbab Center for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands Email:
[email protected]
Abstract In this paper, we examine the inherent properties of some of the most popular coordination models of today and show how they contribute to the difficulty of systematic construction of coordination protocols for the cooperation of concurrent processes, as explicit, tangible pieces of software. They include the object oriented models of communication and the generative tuple space paradigm of models such as Linda. We then describe a new generic model: Idealized Worker Idealized Manager (IWIM) and discuss its advantages for coordination of concurrent activities, especially in controloriented applications. We demonstrate its “completeness” by showing that it can trivially emulate other well-known communication and coordination models. Separation of computation and coordination code into different program modules is one of the important properties of this model, which is fully exploited in the pure coordination language MANIFOLD.
1. Introduction A variety of interesting models, languages, and systems have been proposed for coordination. The interest in this field has recently grown and is further intensified by the expectation to tackle what has been described by some as the next paradigm shift in software[35]. However, not all coordination models, languages, and systems are “created equal” and their use in an application influences its design and its program structure in significant ways. In this respect, we can distinguish three focus points among the expanding body of work on software coordination models, languages, and systems: 1. Some coordination models are used as a means of separating the control issues from the computation concerns in the design and development of programs derived from high-level specifications. This way, the
correctness concerns (i.e., the computation) and efficiency issues (i.e., the coordination) can be dealt with separately[14]. 2. Some coordination models are used as a means to separate the uniform operations on primarily passive data that are common in a large number of parallel/distributed applications, away from the application code, into a small set of generic primitives with their own independent well-defined semantics. The applications most directly amenable to this approach are the data-oriented applications. 3. Some coordination models can be used as a means to separate communication protocols necessary for the cooperation of the active entities that comprise a parallel/distributed application into independent coordination modules. The very large class of applications most directly amenable to this approach are the control-oriented applications. The natural description of the activities and the history of the computation carried out in a data-oriented application tends to center around the data; the application is essentially concerned with what happens to the data. Examples of data-oriented applications include database applications and transaction systems such as banking and airline reservation systems. There is a larger class of applications, called process-oriented or control-oriented where it is unnatural to view their activities and their history as centered around the data. Indeed, often, the very notion of the data, as such, simply does not exist in these applications. For instance, there is no central body of data in a compiler, around which the compilation process can be naturally described in terms of a collection of activities that transform this data; it is more natural to view a compiler as a collection of activities that genuinely consume their input data, and subsequently produce, remember, and transform new data. The above characterizations are not meant as a classification of coordination models and languages. Not only the border lines between the three approaches, above, are
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
blurred, most coordination models and languages are indeed capable of accommodating all three. Nevertheless, these distinctions are useful to show the degree of the appropriateness of a specific coordination model or language for a specific application, by hinting at the natural (mis)match between the underlying paradigm of the former with a specific approach considered for the development of the latter. The use of coordination as a means to separately tackle the correctness and the efficiency aspects of a program (e.g., in [14]) shares similar concerns with formal specifications, automatic derivation of programs from higher-level specifications, and functional programming. A good deal of the existing coordination models and languages (e.g., Linda[12] and Gamma[8]) are based on the notions of shared data spaces that are formally modeled as multisets. They have obvious conceptual similarities with relational databases and with blackboard systems in AI[29]. Comparatively, less work has been reported on models and languages for coordination with specific focus on control-oriented applications. Most of such relevant work takes the message passing paradigm as its base and modifies it by introducing such additional notions as synchronizers, contracts, constraints, and events. A major drawback of these approaches to control-oriented coordination models and languages is that they cannot overcome the deficiencies that are inherent in their underlying message passing paradigm (see x2). The lack of appropriate coordination models and languages is nowhere more obvious (and nowhere more damaging) than in the life cycle of controloriented concurrent (e.g., parallel and/or distributed) applications, which consist of a (large) number of cooperating active entities (processes). The rest of this paper is organized as follows. In x2 we show the shortcomings of the message passing paradigm as a basis for coordination models and languages for concurrent applications. A brief review of some data-oriented coordination models, primarily Linda, is presented in x3. A new model for control-oriented coordination of concurrent activities is discussed in x4. One of the advantages of this model is its separation of computation concerns from communication and coordination responsibilities. An example in x5 demonstrates this point. In x6 we present a concrete and complete pure coordination language based on the model explained in x4, and briefly summarize our experience with this language. Short comparisons of the model described in x4 with message-passing-based and with data-oriented coordination models are presented in x7 and x8, respectively. The conclusion of the paper appears in x9.
2. Where Message Passing Fails The very popular message passing paradigm can be considered as a (low-level) coordination model. It abstracts
away the communication concerns from the computation concerns, into an abstract set of message passing primitives whose semantics can be defined independently of any computation primitives. Higher-level coordination models have been proposed that enhance (various flavors of) the low-level message passing models by additional concepts such as synchronizers, contracts, constraints, rules, and events[17, 6, 20, 23, 2]. It is instructive to first consider the characteristics of a generic message passing model from the view point of coordination concerns and observe its shortcomings. A common characteristic of most flavors of the message passing model is the distinction between the roles they assign to the two active entities involved in a communication: the sender and the receiver. A sender s typically sends a message m to a receiver r. The identity of r is either statically known to s or it is dynamically evaluated at execution time. Sometimes, there is more than one receiver, i.e., the message m is multi-cast to a number of receivers, or it may even be broadcast to all active entities running in an application. One way or the other, the send operation is generally targeted to a specific (set of) receiver(s). A receiver r, on the other hand, typically waits to receive a message m from any sender, as it normally has no prior knowledge of the origin of the message(s) it may receive. We use the term Targeted-Send/Receive, or TSR, to refer to the communication models that share this characteristic. This encompasses theoretical models such as CSP[22]1 , and the Actor model[1]; programming languages such as Occam[25]2 , LOTOS[10], and various flavors of concurrent Object-Oriented and object-based languages; remote procedure call schemes; and concurrent programming tools such as PVM[18], MPI[33], P4[11], PARMACS[21], etc. Consider the following simple example of a concurrent application where the two active entities (i.e., processes) p and q must cooperate with each other. The process p at some point produces two values which it must pass on to q . The process q , in turn, must perform some additional computation using the input it receives from p, and then pass on the result of this computation back to p. Note that it is perfectly meaningful to talk about the cooperation model of this application, independent of the actual processes involved, i.e., a specific p or q , or the computation they perform – as a matter of fact, this is exactly what we just did. The source code for this concurrent application looks something like 1 In fact, the original CSP is stricter than TSR: every receive must specify a source and it will succeed only if the synchronous message it receives is from the source it expects. This inflexibility makes programming in CSP very awkward. For our purposes, however, we may still consider CSP as a special (more rigid) case of a TSR model. 2 Strictly speaking, unlike its underlying CSP model, there are no send and receive operations in Occam; there are only named synchronous channels. However, these channels are statically defined and reading and writing from and to these channels can be thought of as no more than syntactic sugar for send and receive operations.
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
the following: process p:
process q:
compute m1 send m1 to q compute m2 send m2 to q do other things receive m compute with m
receive m1 let z be the sender of m1 receive m2 compute m using m1 and m2 send m to z
The first thing to notice in the above listing is that it is simultaneously both a description of what computation is performed by p and q , and a description of how they cooperate with each other. The communication concerns are mixed and interspersed with computation. Thus, in the final source code of the application, there will be no isolated piece of code that can be considered as the realization of its cooperation model, such that, e.g., we can use it as a module in another application where two other processes are to cooperate with each other in a similar fashion. The second significant point to note in the above listing is the asymmetry between send and receive operations. Every send must specify a target for its message, whereas a receive can receive a message from any anonymous source. In our example, p must know q , otherwise, it cannot send a message to it. The proper functioning of p depends on the availability of another process in its environment that (1) must behave as p expects (i.e., be prepared to receive m1 and m2), and (2) must be accessible to p through the name q . On the other hand, p does not (need to) know the source of the message it receives as m. The assumptions hard-wired into q about its environment (i.e., availability and accessibility of other processes in the concurrent application) are weaker than those in p. The process q waits to receive a message m1 from any source, which it will subsequently refer to as z ; expects a second message m2 (which it can verify to be from the same source, z , if necessary); computes some result, m; and sends it to z . The behavior of the process p, on the other hand, cannot be described without reference to q . The weaker dependence of q on its environment, as compared with p, makes it a more reusable process that can perform its service for other processes in the same or other applications. Note, however, that q is not as flexible as we may want it to be: the fact that the result of its computation is sent back to the source of its input messages is something that is hard-wired in its source code, due to its final targeted send. If, perhaps in a different application environment, we decide that the result produced by q is needed by another process, y, instead of the same process, z , that provides it with m1 and m2, we have no choice but to modify the source code for q . This is a change only to the cooperation model in the
application, not a change to the substance of what q does. The unfortunate necessity of modification to the source code of q , in this case, is only a consequence of its targeted send. An assortment of schemes and constructs have been proposed for coordination based on various incarnations of the message passing paradigm. They include contracts[23], synchronizers[17], constraints[6], constraints and events[34], rules[2], and special protocols[9]. None of these enhancements ameliorate the basic problems of the send-receive asymmetry and the intermixing of computation and communication concerns that are inherent in the generic TSR model.
3. Data-oriented Coordination: Linda and her Friends One of the best known coordination languages is Linda[12]. Linda uses a so called generative communication model, based on a shared tuple space[19]. The tuple space of Linda is a centrally managed data space which contains all pieces of information that processes want to communicate. A process in Linda is a black box. The tuple space exists outside of these black boxes which, effectively, do the real computing. Linda processes can be written in any language. The semantics of the tuples is independent of the underlying programming language used. There are only four communication primitives provided by Linda, each of which associatively operates on a single tuple in the tuple space. The primitive in searches the tuple space for a matching tuple and deletes it; out adds a tuple to the tuple space; rd searches for a matching tuple in the tuple space; and eval starts an active tuple (i.e., a process). The tuple search primitives in and rd block their executing process, if necessary, until they find a suitable tuple in the tuple space. The Linda primitives alleviate one of the problems with the TSR model, by providing symmetric, anonymous means of communication between communicating parties. However, Linda is not a programming language; it is a language extension meant to augment regular programming languages which are to be used to express normal computation. Linda extensions to many languages exist, e.g., C-Linda, PascalLinda, Fortran-Linda, etc. As such, it is not possible to write a “pure Linda program” module that can be compiled separately. Consequently, program modules in a Linda application consist of computation code (e.g., in C) sprinkled with Linda primitives. Just as the case of the TSR model, computation and communication concerns are intermixed in the same source code. Passive and active objects are different entities in Linda which cannot be subjected to the same operations. The essence of the functionality of Linda operations is in their ability to manipulate tuples. This makes Linda directly suitable for certain data-oriented applications. However, the
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
semantics of Linda primitives and the existence of a single global tuple space makes it less trivial to use in many other applications. Even certain “database-type” applications are not so easily expressed using Linda primitives. For instance, consider what has been called the “multiple read” problem, an example of which can be seen in a program for computing the join of two relations as in a relational database. Suppose each relation is expressed as a set of tuples in the tuple space. Intuitively, one would expect it to be straight-forward to write a Linda application that uses n processes to concurrently perform the join (over a common attribute). However, the semantics of the rd operation is inadequate for this task; in fact, what is needed here is the ability for a process to start with its own unique scan of each relation and obtain each tuple in a relation exactly once in its scan. The Linda rd operation does not support the notion of a scan, and, indeed, several rd operations by a process can return the very same tuple. Proper solutions to this problem require either new Linda operations, or resorting to programming tricks involving locks, additional tags in tuples, etc., none of which is simultaneously elegant, simple, and efficient[32]. More generally, if more than one process can obtain a tuple from the tuple space, a protocol is needed to avoid undesirable effects. This presents a problem in reflecting the natural hierarchy of applications (such as divide-andconquer or domain decomposition) that involve independent groups of processes, in their implementation. Ideally, each such group should communicate through its own private medium, to avoid accidental conflicts among the communications of unrelated parties. However, secure, private communication between two processes is not directly possible in Linda. Folding all these private communication media into one shared tuple space involves some non-trivial protocols and programming tricks. The deficiency of Linda with regards to safe communication among a set of processes has been noted by several researchers. A variation of Linda that supports safe, direct communication (akin to message passing) between pairs of processes is proposed in [31]. Several researchers have proposed multiple tuple spaces as a means to remedy this deficiency (in addition to other benefits that they bring)[26]. Another approach to remedy this deficiency and, simultaneously, enforce other constraints is presented in Law-Governed Linda[28]. In Law-Governed Linda, laws regulate the interactions of individual processes with the shared tuple space (and therefore with each other), analogous to the manner in which social laws, e.g., secure financial transactions in the market place[28]. In effect, laws in Law-Governed Linda establish various forms of secure message passing as well as multiple tuple spaces. Every process in Law-Governed Linda has a controller that acts as the mediator between it and the tuple
space to ensure its compliance with the laws of the system. The laws are expressed in a restricted version of Prolog. Bauhaus Linda is a generalization of Linda where the notions of tuple and tuple space are unified into the single notion of a multiset[13]. This generalization both simplifies Linda and, simultaneously, makes it more powerful and expressive. Bauhaus Linda makes no distinction between passive and active objects. The concept of multisets allows multiple tuple spaces, as well as protected, safe, private communication. It is possible to have a hierarchy of multiple tuple spaces in Bauhaus. However, as Linda, Bauhaus is not a complete language: it is an extension that must still be embedded into a sequential computation language before it can be used. This means that, as in Linda, Bauhaus does not enforce a separation of communication and computation concerns. The intuitive simplicity of Linda is appealing. However, many real applications require more complex coordination constructs that are not always easily mapped into the simple operations of Linda. The fact that many such constructs can actually be expressed as protocols that can be programmed in Linda, shows the versatility of the Linda model. Unfortunately, it also means that the conceptual models one must deal with in such applications often involve non-trivial protocols that are a far cry from the underlying simple model of Linda used to implement them.
4. Control-oriented Coordination: The IWIM Model We may regard each process as an individual worker. When the TSR model is used, a worker must, naturally, know how to produce the (partial) results it is expected to produce, and it also must either (1) know by name the workers it must deliver its results to, or (2) know how to find out the identities of those workers, e.g., know someone who knows them. The Idealized Worker Idealized Manager (IWIM) model[4, 3] is an alternative generic model for controloriented coordination of concurrency that, unlike the TSR model, supports the separation of responsibilities and encourages a weak dependence of workers on their environment. Like the TSR model, the IWIM model is described only in terms of its most significant characteristics. As such, like the TSR model, it indeed defines not a specific model of communication, but a family of such models. Various members in this family can have different significant characteristics, e.g., with regards to synchronous vs. asynchronous communication.
4.1. Basic Concepts The basic concepts in the IWIM model are processes, events, ports, and channels. A process is a black box with
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
well defined ports of connection through which it exchanges units of information with the other processes in its environment. A port is a named opening in the bounding walls of a process through which units of information are exchanged using standard I/O type primitives analogous to read and write. Without loss of generality, we assume that each port is used for the exchange of information in only one direction: either into (input port) or out of (output port) a process. We use the notation p:i to refer to the port i of the process instance p. The interconnections between the ports of processes are made through channels. A channel connects a (port of a) producer (process) to a (port of a) consumer (process). We write p:o q:i to denote a channel connecting the port o of the producer process p to the port i of the consumer process q.
!
Independent of the channels, there is an event mechanism for information exchange in IWIM. Events are broadcast by their sources in their environment, yielding an event occurrence. In principle, any process in an environment can pick up a broadcast event occurrence. In practice, usually only a few processes pick up occurrences of each event, because only they are tuned in to their sources. The IWIM model supports anonymous communication: in general, a process does not, and need not, know the identity of the processes with which it exchanges information. This concept reduces the dependence of a process on its environment and makes such processes more reusable. A process in IWIM can be regarded as a worker process or a manager (or coordinator) process. In general, no process in IWIM is responsible for its own communication with other processes. It is always the responsibility of a manager process to arrange for and to coordinate the necessary communications among a set of worker processes. Each process receives the input it requires at its own input ports. Its mandate is (to manage other processes, if necessary) to compute the results it is expected to produce, and to deliver them at its own output ports. Generally, how its proper input gets to its input ports, how the output it produces is delivered to other processes, and the identity of the producers and the consumers of its input and output are of no concern to a process. There is always a bottom layer of worker processes, called atomic workers, in an application. In the IWIM model, an application is built as a (dynamic) hierarchy of (worker and manager) processes on top of this layer. Aside from the atomic workers, the categorization of a process as a worker or a manager process is subjective: a manager process m that coordinates the communication among a number of worker processes, may itself be considered as a worker process by another manager process responsible for coordinating the communication of m with other processes.
4.2. Communication Channels A channel is a communication link that carries a sequence of bits, grouped into (variable length) units. A channel represents a reliable, directed, and perhaps buffered, flow of information in time. Reliable means that the bits placed into a channel are guaranteed to flow through without loss, error, or duplication, with their order preserved. Directed means that there are always two identifiable ends in a channel: a source and a sink. Once a channel is established between a producer process and a consumer process, it operates autonomously and transfers the units from its source to its sink. With no assumptions about the internal operation of the producer and the consumer of a channel C , it is possible for C to contain some pending units. Therefore, it may or may not be desirable for a channel to immediately disconnect itself from its source or its sink as soon as its connection at its opposite end is broken. The pending units of a channel C are those that have already been delivered to C by its producer, but not yet delivered by C to its consumer. The possibility of the existence of pending units in a channel gives it an identity of its own, independent of its producer and consumer; e.g., it makes it meaningful for a channel to remain connected at one of its ends, after it is disconnected from the other. There are five different types of channels in the IWIM model, a synchronous channel (S) and four asynchronous channels (BB, BK, KB, and KK3 ): 1. S channel: There is never a pending unit in a channel of this type. It is meaningless to talk about a channel of this type without a complete producer-consumer pair. 2. BB channel: A channel of this type is disconnected from either of its producer or consumer automatically, as soon as it is disconnected from the other. 3. BK channel: A channel of this type is disconnected from its producer automatically, as soon as it is disconnected from its consumer, but disconnection from its producer does not disconnect the channel from its consumer. 4. KB channel: A channel of this type is disconnected from its consumer automatically, as soon as it is disconnected from its producer, but disconnection from its consumer does not disconnect the channel from its producer. 3 The letters “B” and “K” in the name of a channel type, respectively designate break and keep type connections to the ports at their corresponding ends of the asynchronous channels of that type.
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
5. KK channel: A channel of this type is not disconnected from either of its processes automatically, if it is disconnected from the other.
4.3. Primitives for Workers and Managers There are two means of communication available to a process in IWIM: via its ports, and via events. The communication primitives that allow a process to exchange units through its ports are analogous to the traditional read and write I/O primitives. A process p can broadcast an event e to all other processes in its environment by raising that event. The identity of the event e together with the identity of the process p comprise the broadcast event occurrence. A process can also pick up event occurrences broadcast by other processes and react on them. Any process in IWIM can raise an event, react on event occurrences it has detected, and read from and write to its own ports. Only manager processes can (dynamically) create process instances and channels, and (re)connect channels to the ports of (their own or other) processes; and that is all they can do: managers have no primitives to enable them to carry out any computation themselves. Each manager process typically controls the communications among a (dynamic) number of process instances in a data-flow like network. The processes themselves are generally unaware of their patterns of communication, which may change in time, by the decisions of their respective manager processes.
5. Communication vs. Computation Wegner proposes extending the Turing machine by adding input and output actions to its set of primitive operations[36]. A Turing machine can only operate in batch mode, i.e., it can exchange information with its environment only though its tape, and only before and after it performs its computation. Unlike the standard read and write actions that operate on its tape, the new input/output actions enable an extended Turing machine to exchange information with its environment interactively, i.e., while it is carrying out a computation. Wegner calls such an extended Turing machine an interaction machine and argues that interaction machines can carry out computation beyond the power of Turing machines. One of the significant differences between the two is in the interaction machines’ ability to comprise open systems. Anonymous communication in IWIM supports open system architectures and interaction. The fact that all that an idealized manager can accomplish is possible only through interactions it facilitates using the “coordination primitives” parallels some of the arguments of Wegner. The notion that computation can be constructed out of nothing but communication is also at the foundation of -calculus[27].
Our example of x2 expressed in the IWIM model consists of three processes: revised p, revised q , and a coordinator process c which is responsible to facilitate their communication. The source code for this version of the application looks something like the following: process p:
process q:
compute m1 write m1 to port o1 compute m2 write m2 to port o2 do other things read m from port i1 compute with m
read m1 from port i1 read m2 from port i2 compute m using ... ... m1 and m2 write m to port o1
process c: create the channel p.o1 create the channel p.o2 create the channel q.o1
!q.i1 !q.i2 !p.i1
In this example, the pattern of cooperation between the processes p and q is simple and static. Therefore, the responsibility of the coordinator process c is, indeed, very simple: perhaps, it first creates the processes p and q , establishes the communication channels defined above, and then may wait for the proper condition (e.g., termination of p and/or q ) to dismantle these channels and terminate itself. Nevertheless, moving the communication concerns out of p and q and into c already shows some of the advantages of the IWIM model. The processes p and q are now “ideal” workers. They do not know and do not care where their input comes from, nor where their output goes to. They know nothing about the pattern of cooperation in this application; they can just as easily be incorporated in any other application, and will do their job provided that they receive “the right” input at the right time. The cooperation model of this application is now explicit: it is embedded in the coordinator process c. If we wish to have the output of q delivered to another process, or to have yet another process deliver the input of p, neither p nor q , but only c is to be modified. The process c is an “ideal” manager. It knows nothing about the details of the tasks performed by p and q . Its only concern is to ensure that they are created at the right time, receive the right input from the right sources, and deliver their results to the right sinks. It also knows when additional new process instances are supposed to be created, how the network of communication channels among processes must change in reaction to significant event occurrences, etc. (none of which is actually a concern in this simple example). It is very likely that such ideal worker processes developed for one application can be used in other concurrent
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
applications, with very different cooperation patterns. Removing the communication concerns out of worker processes enhances the modularity and the re-usability of the resulting software. Furthermore, the fact that such ideal manager processes know nothing about the tasks performed by the workers they coordinate, makes them generic and reusable too. The cooperation protocols for a concurrent application can be developed modularly as a set of coordinator processes. It is likely that some of such ideal managers, individually or collectively, may be used in other applications, coordinating very different worker processes, producing very different results; as long as their cooperation follows the same protocol, the same coordinator processes can be used. Modularity and re-usability of the coordinator processes also enhance the re-usability of the resulting software[5].
6. Manifold MANIFOLD is a coordination language based on the IWIM model for managing complex, dynamically changing interconnections among sets of independent, concurrent, cooperating processes[3]. A MANIFOLD application consists of a (potentially very large) number of (light- and/or heavyweight) processes running on a network of heterogeneous hosts, some of which may be parallel systems. Processes in the same application may be written in different programming languages. Some of them may not know anything about MANIFOLD, nor the fact that they are cooperating with other processes through MANIFOLD in a concurrent application. MANIFOLD is the first pure coordination language based on the IWIM model. However, it is not the only possible incarnation of IWIM. Another example of a coordination system in the IWIM family is described in [24]. MANIFOLD enforces IWIM’s clean separation of computation and coordination code into isolated modules. MANIFOLD does not provide any computation constructs and its interface to other languages provides no coordination primitives. Thus, only computation code can be written in other languages used in a MANIFOLD application, and its (separately compiled) MANIFOLD code will consist of coordination modules only. This results in highly modular and reusable computation components, as well as highly modular and reusable coordination components; the latter is a novel and, so far, unique property of MANIFOLD applications[5]. Examples of MANIFOLD for the solution of domain decomposition problems[15], restructuring of existing sequential code into parallel/distributed applications[16], and emulating the constructs of soft real-time languages[30] are discussed elsewhere.
7. IWIM and TSR The TSR model can be regarded as a special restricted case of the IWIM model, where only senders of messages are permitted to establish communication channels. Thus, for instance, every send operation first verifies to see if there is a communication channel between the sender and its target process. If such a channel exists, the message is simply placed into the channel. If such a channel does not exist, the send operation first creates the channel, and then proceeds to place its message into the channel. Using the IWIM model to describe a typical TSR communication system allows a more precise and explicit definition of the semantics of the latter’s communication primitives. For example, PARMACS[21] and MPI[33] support both synchronous and asynchronous communication, whereas PVM[18] supports only asynchronous message passing. The synchronous communication send and receive primitives in these packages correspond to the S-type channel in IWIM. A closer examination of the behavior of the asynchronous send and receive primitives in these packages, reveals that they actually correspond to different channel types in IWIM. In PARMACS and MPI, once an asynchronous message is sent, its receiver can still receive it even after its sender process has terminated. In the IWIM terminology, once an asynchronous channel of communication is established, the messages placed into the channel are still available for the sink process, even after the source process terminates. However, an asynchronous send in PARMACS and MPI that specifies a non-existent target process fails and results in an error code. In the IWIM terminology, this means that the asynchronous channel of communication between a source and a sink will break from its source as soon as its sink terminates. Thus, the behavior of the asynchronous communication primitives in PARMACS and MPI correspond to the BK-type channel in IWIM. On the other hand, the asynchronous communication in PVM is different. As in the case of PARMACS and MPI, pending messages sent by a terminated sender can still be received by a receiver at its leisure. However, it is indeed possible for a sender in PVM to continue sending messages to a non-existent target. As far as the sender is concerned, all such send operations succeed. In the IWIM terminology, this means that the connection between a source and a channel is not severed just because its sink process has terminated. Thus, the asynchronous communication in PVM corresponds to the KK-type channel in IWIM. The difference between the asynchronous communication in PARMACS and MPI on the one hand, and PVM, on the other, is a significant aspect of their semantics which substantially affects the communication protocols of applications developed using these systems. Nevertheless, such
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
detail is rather obscure in the documentation of these packages and these distinctions rarely receive the proper attention they deserve in the formal work on communication and concurrency. The four different asynchronous channel types in IWIM make it explicitly clear that it is never sufficient to describe a communication channel only as asynchronous: at least enough additional information must be supplied to specify one of the four possible IWIM alternatives. Different sets of constructs for coordination have been proposed to augment the basic message passing paradigm, yielding an assortment of coordination models, language extensions, and schemes. These proposals constitute various incarnations of the TSR model. What follows is a brief review of some of these schemes, and of how they relate to IWIM. The main construct offered in [17] is a synchronizer, which is to be integrated into an object oriented concurrent language that adheres to the Actor model[1] of computation. Coordination patterns can be expressed as constraints that restrict invocation of a group of objects. These constraints are defined in terms of the interface of the objects being invoked, rather than their internal representation. Such invocation constraints enforce properties, such as temporal ordering and atomicity, that must hold when objects are invoked in a group. Through invocation constraints, coordination patterns can be specified abstractly, independent of the protocols needed to implement them. Enforcement of constraints is done by synchronizers, which are special objects that observe and limit the invocations accepted by a set of ordinary objects which are being constrained. This is somewhat similar to the idea of workers and managers in the IWIM model. As with managers in IWIM and manifolds in MANIFOLD, synchronizers can overlap: multiple separately specified synchronizers can constrain the same objects. Synchronizers themselves are not “real objects” in the sense that it is not possible to, recursively, constrain their behavior using other synchronizers, and ordinary objects cannot send messages to them as they do to other ordinary objects. In IWIM (and MANIFOLD) a manager (manifold instance) is externally indistinguishable from an ordinary worker (atomic process). Synchronizers enforce constraints that can express certain abstract high-level coordination concerns only. As the authors emphasize, the concept of synchronizers is not the complete answer to the challenge of coordination[17]. Among other shortcomings, only “reactive” behavior can be described through constraints. In IWIM, coordination can be expressed explicitly (in MANIFOLD, this is done as state diagrams), which allows for “pro-active” coordination behavior. In an Actor-based language extended with synchronizers, as proposed in[17], the basic model of communication is a variant of the (asynchronous) TSR model, wherein the
computation and communication concerns can be mixed together in the same modules. In contrast, there is a clear separation of computation and communication concerns in IWIM (and MANIFOLD). Dragoon[6] is an object oriented programming language that allows specification of synchronization constraints externally to an an object. Unlike the synchronizers in[17], Dragoon allows specification of coordination of single objects only. Procol[34] augments the notions of constraints and events to an object oriented language. Events can trigger constraints which can reactively observe invocations (message passing), but, unlike the synchronizers in[17], they cannot inhibit or limit them. The communication models of Procol and Dragoon fit in the TSR family, rather than IWIM. A model for pro-active coordination based on intensional interpretation of declarative rules in an object oriented paradigm is described in [2]. Rules are, by themselves, declarative statements. The entities that constitute the domain of discourse for the coordination rules (tokens) are messages. A rule-based coordinator interprets the rules and engages groups of participant objects in a dialog through meta-transactions. This requires participant objects to be fully prepared to engage in a non-trivial, uniform protocol of interaction with the coordinator. Participants communicate with each other through message passing (i.e., following the TSR model) and coordinators monitor, interpret, and intervene in this communication. There is a clear distinction between participants (objects) and coordinators in this model; a coordinator is an active entity whose behavior is described by the declarative coordination rules. It is not clear if an application can have more than one coordinator, and if so, what happens if there is a conflict between the sets of rules in different coordinators. The different “nature” of coordinators vs. participants makes it difficult to conceive of meta-level coordinators in this model. On the other hand, because in IWIM and MANIFOLD, a coordinator cannot distinguish between another coordinator and an ordinary worker process, a hierarchy of coordinators is possible. The concept of contracts, introduced in[20, 23] extends the notion of type to capture the message passing obligations of objects. Contracts are intuitively similar to constraints imposed on object behavior as in[17], except that contracts are descriptive as opposed to the prescriptive nature of the synchronizers of [17]. A molecular dynamics simulation program implemented in a concurrent object oriented language is described as a case study of a complex parallel application in [9]. Through this case study, its authors show the necessity for two kinds of coordination protocols, which they call sender-initiated and receiver-initiated coordination. Their sender-initiated protocol essentially involves a multi-cast, which can be supported in a number of languages and models, including IWIM and MANIFOLD. Their receiver-initiated protocol involves
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
synchronization of an object based on the state of other objects. It is non-trivial to achieve this type of coordination in many languages and models. The authors mention the dining philosophers problem as an example to demonstrate the general necessity for this type of synchronization. Synchronizing a process based on the (reported) internal state of another process is straight forward in IWIM and MANIFOLD. A process can report its internal states by raising events. It is easy to program the dining philosophers problem in MANIFOLD using this scheme.
8. IWIM and Linda
Conceptually, IWIM and Linda are very different models for coordination. At some fundamental level, they can be considered to be “equivalent” in the sense that each can emulate the other. They both provide symmetric primitives for anonymous communication. Linda provides a single shared data-space as the universal medium of (asynchronous) communication among all processes, whereas IWIM provides (both synchronous and asynchronous) private channels. One major difference between the two is that Linda is dataoriented, whereas IWIM is a control-oriented model. The main advantage of IWIM is that it supports a clean separation of computation and coordination concerns into different program modules (as in MANIFOLD), and encourages the decomposition of both computation and coordination tasks into hierarchies of small, reusable pure computation modules and small, reusable pure coordination modules. The software engineering advantages inherent in the IWIM model make it worthwhile to consider an implementation of IWIM on top of Linda, yielding IWIM-Linda. Such an
implementation is non-trivial. Essentially, it involves overcoming and programming around most of the very same deficiencies in Linda faced by other researchers and practitioners reported in the literature. Some, but not all, of these programming contortions can be avoided if instead of Linda, some other multiset-based coordination model (e.g., Bauhaus Linda[13] or Gamma[7]) is used. On the other hand, the simplicity of Linda (and most other multiset-based models) and the expressiveness of IWIM make emulation in the opposite direction a straight-forward task. For instance, the shared tuple space of Linda can be implemented as a single global process in IWIM that provides a pair of communication ports for each individual process in a Linda application. The Linda primitive operations can then use these communication ports to exchange tuples with this global process, which internally, maintains the contents of the Linda data space.
9. Conclusion Modularity, weaker dependence of modules on their environment, and separation of the orthogonal concerns for communication and coordination on the one hand, and computation on the other, are important aspects of a model for coordination of concurrent activities, that have drastic practical consequences in software engineering. Coordination schemes that are based on the message passing paradigm suffer from two basic drawbacks when they are used in the construction of large concurrent applications. Although the generative tuple space model of Linda suits the requirements of certain data-oriented concurrent applications, Linda-like schemes, even when extended with the notion of multisets and multiple tuple spaces, become cumbersome to use in control-oriented (and even in some data-oriented) applications. IWIM is a new generic model for coordination of concurrent activities. By avoiding the shortcomings of the other models examined in this paper, IWIM offers a more suitable basis for control-oriented coordination languages. The significant characteristics of the IWIM model include compositionality, which it inherits from data-flow, anonymous communication, and a clean separation of computation concerns from communication concerns. These characteristics lead to clear advantages in construction of large concurrent applications. MANIFOLD is a concrete programming language that is directly based on the IWIM model. Running examples of MANIFOLD programs show that the advantages of the IWIM model do indeed translate well into practice and enhance the modularity and re-usability of both the coordination protocols, as well as the computation components of concurrent applications.
References [1] G. Agha. Actors: A Model of Concurrent Computation in Distributed Systems. MIT Press, 1986. [2] J. Andreoli, H. Gallaire, and R. Pareschi. Rule-Based Object Coordination. In P. Ciancarini, O. Nierstrasz, and A. Yonezawa, editors, Object-Based Models and Languages for Concurrent Systems, volume 924 of Lecture Notes in Computer Science, pages 1–13. Springer-Verlag, Berlin, 1995. [3] F. Arbab. Coordination of massively concurrent activities. Technical Report CS–R9565, Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, November 1995. Available on-line http://www.cwi.nl/ftp/CWIreports/IS/CS-R9565.ps.Z. [4] F. Arbab. The IWIM model for coordination of concurrent activities. In P. Ciancarini and C. Hankin, editors, Coordination Languages and Model, volume 1061 of Lecture Notes in Computer Science, pages 34–56. Springer-Verlag, April 1996.
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
[5] F. Arbab, C. L. Blom, F. J. Burger, and C. T. H. Everaars. Reusable coordinator modules for massively concurrent applications. In L. Bouge, P. Fraigniaud, A. Mignotte, and Y. Robert, editors, Proceedings of Euro-Par ’96, volume 1123 of Lecture Notes in Computer Science, pages 664–677. Springer-Verlag, August 1996. [6] C. Atkinson, S. Goldsack, A. D. Maio, and R. Bayan. Objectoriented concurrency and distribution in DRAGOON. Journal of Object-Oriented Programming, March/April 1991. [7] J. Banatre and D. L. Metayer. The GAMMA model and its discipline of programming. Science of Computer Programming, 15:55–77, November 1990. [8] J. Banatre and D. L. Metayer. Programming by multiset transformations. Communications of the ACM, 36(1):98– 111, January 1993. [9] S. Bijnens, W. Joosen, and P. Verbaeten. Sender initiated and receiver initiated coordination in a global object space. In P. Ciancarini, O. Nierstrasz, and A. Yonezawa, editors, Object-Based Models and Languages for Concurrent Systems, LNCS 924, pages 14–28. Springer-Verlag, 1995. [10] T. Bolognesi and E. Brinksma. Introduction to the ISO specification language LOTOS. Computer Networds and ISDN Systems, 14:25–59, 1986. [11] R. Butler and E. Lusk. User’s guide to the p4 parallel programming system. Technical Report ANL-92/17, Argonne National Laboratory, Oct. 1992. Version 1.4. [12] N. Carriero and D. Gelernter. LINDA in context. Communications of the ACM, 32:444–458, 1989. [13] N. Carriero, D. Gelernter, and L. Zuck. Bauhaus Linda. In P. Ciancarini, O. Nierstrasz, and A. Yonezawa, editors, Object-Based Models and Languages for Concurrent Systems, volume 924 of Lecture Notes in Computer Science, pages 66–76. Springer-Verlag, Berlin, 1995. [14] M. Chaudron and E. de Jong. Towards a compositional method for coordinating Gamma programs. In P. Ciancarini and C. Hankin, editors, Coordination Languages and Model, volume 1061 of Lecture Notes in Computer Science, pages 107–123. Springer-Verlag, April 1996. [15] C. T. H. Everaars and F. Arbab. Coordination of distributed/parallel multiple-grid domain decomposition. In A. Ferreira, J. Rolim, Y. Saad, and T. Yang, editors, Proceedings of Irregular ’96, volume 1117 of Lecture Notes in Computer Science, pages 131–144. Springer-Verlag, August 1996. [16] C. T. H. Everaars, F. Arbab, and F. J. Burger. Restructuring sequential Fortran code into a parallel/distributed application. In Proceedings of the International Conference on Software Maintenance ’96. IEEE, November 1996. [17] S. Frolund and G. Agha. A language framework for multiobject coordination. In Proc. ECOOP ’93, volume 707 of Lecture Notes in Computer Science, pages 346–360. Springer-Verlag, 1993. [18] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam. PVM 3 user’s guide and reference manual. Technical Report ORNL/TM-12187, Oak Ridge National Laboratory, September 1994. [19] D. Gelernter. Generative communication in Linda. ACM Transactions on Programming Languages and Systems, 7(1):80–112, 1985.
[20] R. Helm, I. Holland, and D. Gangopadhyay. Contracts: Specifying behavioral compositions in object-oriented systems. SIGPLAN Notices, 25(10):169–180, October 1990. [21] R. Hempel, H. Hoppe, U. Keller, and W. Krotz. PARMACS v6.1 specification. Technical report, PALLAS GmbH, Hermulheimer Strasse 10, D-50321, August 1995. [22] C. Hoare. Communicating Sequential Processes. Prentice Hall International Series in Computer Science. Prentice-Hall, New Jersey, 1985. [23] I. Holland. Specifying reusable components using contracts. In O. L. Madsen, editor, Proc. ECOOP ’92, volume 615 of Lecture Notes in Computer Science, pages 287–308. Springer-Verlag, July 1992. [24] A. A. Holzbacher. A software environment for concurrent coordinated programming. In P. Ciancarini and C. Hankin, editors, Coordination Languages and Model, volume 1061 of Lecture Notes in Computer Science, pages 249–266. Springer-Verlag, April 1996. [25] INMOS Ltd. OCCAM 2, Reference Manual. Series in Computer Science. Prentice-Hall, London — Sydney — Toronto — New Delhi — Tokyo, 1988. [26] K. M. Kahn and M. S. Miller. A letter about the article ’Linda in Context’ by Carrierro and Gelernter. Communications of the ACM, 32(10):1253–1255, October 1989. [27] R. Milner. Elements of interaction. Communications of the ACM, 36(1):78–89, January 1993. [28] N. Minsky and J. Leichter. Law-governed Linda as a coordination model. In P. Ciancarini, O. Nierstrasz, and A. Yonezawa, editors, Object-Based Models and Languages for Concurrent Systems, LNCS 924, pages 125–146. Springer-Verlag, 1995. [29] H. P. Nii. Blackboard systems. In A. Barr, P. Cohen, and E. Feigenbaum, editors, The Handbook of Artificial Intelligence, volume 4, pages 1–82. Addison-Wesley, 1989. [30] G. A. Papadopoulos and F. Arbab. Coordination of systems with real-time properties in Manifold. In Proceedings of the 20th Annual International Computer Software and Applications Conference (Compsac ’96). IEEE, August 1996. [31] J. Pinakis. Providing direct communication in Linda. In 15th Australian Computer Science Conference, pages 731– 743, 1992. [32] A. Rowstron and A. Wood. Solving the Linda multiple rd problem. In P. Ciancarini and C. Hankin, editors, Coordination Languages and Model, volume 1061 of Lecture Notes in Computer Science, pages 357–367. Springer-Verlag, April 1996. [33] The Message Passing Interface Forum. MPI: A Message Passing Interface Standard, May 1994. Available on-line http://www.mcs.anl.gov/mpi/mpi-report.ps. [34] J. van den Bos and C. Laffra. PROCOL: A concurrent objectoriented language with protocols delegation and constraints. Acta Informatica, 28:511–538, 1991. [35] P. Wegner. Coordination as constrained interaction. In P. Ciancarini and C. Hankin, editors, Coordination Languages and Model, volume 1061 of Lecture Notes in Computer Science, pages 28–33. Springer-Verlag, April 1996. [36] P. Wegner. Foundations of interactive computing. Technical Report CS-96-01, Brown University, April 1996.
Proceedings of The Thirtieth Annual Hawwaii International Conference on System Sciences ISBN 0-8186-7862-3/97 $17.00 © 1997 IEEE
1060-3425/97 $10.00 (c) 1997 IEEE
