Mianjin is Gardens Point: A Parallel Language Taming Asynchronous Communication Paul Roe and Clemens Szyperski Queensland University of Technology, Brisbane, Australia fp.roe,
[email protected]
Abstract. The Gardens language, Mianjin, supports task based parallel
computation, which utilises Active Messages style asynchronous communication. Unlike raw Active Messages, which is a library, this is achieved safely and without degrading performance by enforcing all necessary restrictions statically at the language level. Also supported is the de nition of communication abstractions. To do this Mianjin utilises a partitioned address space where local objects are distinguished from those made globally accessible; this is achieved via type annotations. Type annotations are also used to distinguish those routines which may perform communications from those which will not; a distinction important for the static safety of an Active Message implementation. A special parameter mechanism is used to implement and enforce the conditions necessary for safe reply (read) operations.
1 Introduction This paper describes the Gardens programming language, Mianjin1, and its support for global objects. Gardens is an integrated programming language and system which supports ecient parallel computing across networks of workstations (cluster computing). Note that Mianjin is a general purpose parallel language which is not restricted to just cluster computing. The goals of Mianjin are: { Eciency (with costs visible to programmers) { Support for building communication abstractions { Safety both in terms of memory and communications invariants It is the combination of these points which distinguishes our work from that of others. Mianjin is based on the Oberon programming language [6]: the latest in the Pascal family. Oberon may be characterised as being: small, ecient and safe; it also supports object oriented programming. Mianjin, and the whole Gardens project, has been designed with a minimalist philosophy, inspired by the Oberon programming language and system [9]. Thus Mianjin is designed to provide the basic building blocks required for safe and extensible parallel programming. 1
`Mianjin' means place of the blue water lilies; it is an Aboriginal name for Gardens Point, where QUT is located.
For exibility and performance, Mianjin utilises a task model of parallel computation. In Gardens, tasks are units of work, they are used for load balancing, and may be dynamically created. Thus Gardens utilises multitasking on individual processors; however, this is not a prerequisite for Mianjin. What Mianjin does support directly is the notion of multiple isolated address spaces or address space partitions; therefore communication will be discussed in terms of communication between address spaces. Given an ecient sequential implementation of Mianjin, an ecient parallel implementation requires high performance communications. This is achieved via a very lightweight and ecient messaging software layer: Active Messages[8] (AM)2 . Unfortunately, although AM provides performance it is subject to a number of restrictions which conventional languages, such as C, cannot enforce. Furthermore, AM does not lend itself to the formation of abstractions. Typically communication either occurs through global variables, which do not support abstraction; or some kind of unenforced, unsafe programming conventions must be adopted; or a more complex, and less ecient, communications layer is built on top of AM. Mianjin supports AM via global objects and associated methods which enable abstractions to be programmed via object encapsulation and polymorphism. Remote method invocations are realised using AM, and without incurring any signi cant overhead. Furthermore the language statically guarantees the safe use of global objects via its type system and in particular its type annotations. Thus Mianjin raises the level of abstraction of AM and guarantees safety without sacri cing performance. This paper is organised as follows: the next section describes AM and its restrictions. Section 3 describes the partitioning of address spaces via local and global objects. Section 4 describes atomicity restrictions in the form of type annotations which enforce safe AM operations and support abstraction. Section 5 describes how reply/read operations are supported via a special parameter passing mechanism and reply statements. Section 6 summarises the type annotations for di erent kinds of procedures and methods, and Section 7 overviews the implementation of Mianjin. The last sections describe related and future work.
2 The Base: Active Messages Active Messages represents the state-of-the-art for fast messaging on parallel computers [8]. In essence, AM is a form of lightweight asynchronous remote procedure call. The AM design assumes a few strong invariants that guarantee the following properties: 1. Non-preemptive semantics. 2. Local AM calls almost as ecient as local procedure calls. 3. No self-in icted deadlock despite non-preemptive semantics. 2
We are using AM version 1.0; a more sophisticated version, AM 2.0, is under development at UC Berkeley.
4. No distributed network deadlock caused by bu er over ow. These are very important properties of AM, and thus Mianjin statically guarantees the AM invariants to support these. The properties are described in more detail below. In AM, request operations may be issued which asynchronously send a message to a remote processor, consisting of a handler (function pointer) and some data. On receipt messages are queued until a poll operation is performed. \Poll" processes messages by invoking handlers on associated data; this gives the recipient control over when handlers are invoked (Property 1). Note that \poll" may process any message; it is not possible to lter particular messages. \Poll" does guarantee that messages are processed atomically, and in order, if from the same sender. A request handler may invoke a reply operation (similar to a request) to return a message to the original sender; however reply handlers may perform no communication. Thus communication operations cannot be nested via handlers. If the destination of a request (or reply) is the local processor the addressed handler will be called immediately,approaching the eciency of a local procedure call (Properties 2 and 3). Thus all request operations also perform a poll on completion; this ensures the semantics of local requests is consistent with that of non-local requests. A credit counting scheme is used to control network deadlock (Property 4): as opposed to application deadlock. Credits are lost by sending messages to a host, and gained by receiving reply/acknowledgement messages from hosts. Therefore, no protocol is needed to handle bu er over ow at the receiving end; in general, AM's performance is largely a result of trimming back traditional network and transport protocol overheads. If a request is issued when the credit count is zero the request operation will poll until credit is available; after which the request will be performed. Thus, a request operation may cause a poll before and after its operation. In fact, requests (and replies) are for short communications using messages short enough to be passed in registers on many machines; their longer variants, called \stores", are discussed in Section 7. All discussion is initially in terms of request and reply.
3 Global Objects In order to tame AM|make it safe, support abstraction and maintain performance| Mianjin supports global objects (GOs). Essentially, Mianjin permits safe AM like asynchronous communication between di erent address spaces via GOs. AM's remote procedure calls are lifted to remote (global) method invocations. Like several other proposals Mianjin partitions its address space: typically between tasks. All objects inhabit a particular address space; some may be only locally accessible, others (global objects) are accessible from all address spaces. A key idea in Mianjin is that only global references to global objects and copied ( at) values may be communicated between address spaces; in particular local
pointers cannot escape an address space. Note, Mianjin does not support global variables. Global objects are denoted by the \GLOBAL" type annotation. Since AM is asynchronous all communication must occur through heap allocated objects in order for it to be safe. (A stack allocated object could be out of scope at the time of message arrival.) For example a program may create an ordinary local object thus: VAR o: POINTER TO Object; ... NEW(o);
(Note in Mianjin objects such as Object are simply records.) Local objects are compatible with global ones, but not vice versa; for example a local object, more speci cally a reference to it, can be assigned to a global one thus: VAR o: POINTER TO Object; g: GLOBAL POINTER TO Object; ... g := o;
Global objects are local objects, potentially made globally accessible, but subject to certain restrictions: { they must be heap allocated (ie of type GLOBAL POINTER TO T), { they cannot be dereferenced (their elds are not directly accessible) and { only methods of restricted signature can be invoked on them3 An ordinary method, for a local object, can be declared thus: TYPE Accum = POINTER TO RECORD count: INTEGER; sum: REAL END; PROCEDURE (self: Accum) Add (v: REAL); BEGIN self.sum := self.sum + v END Add;
it may be invoked thus: VAR o: Accum; ... o.Add(3);
A global method is denoted by preceding its name with the keyword GLOBAL: GLOBAL PROCEDURE (self: Accum) GAdd (v: REAL); BEGIN self.sum := self.sum + v; INC(self.count) END GAdd;
3
However all methods are available to local references to objects.
This method adds a value to the sum eld of a GO and counts the number of such additions. It can be used to implement an accumulator (sum reduction) abstraction. The global method may be invoked thus: VAR a: GLOBAL Accum; ... a.GAdd(4);
Method invocations on global objects are transmitted to the local address space of the global object, and invoked locally; thus they have access to object (record) elds. Such global method invocation on GOs is implemented by AM request/store operations, and therefore may cause implicit polling (see previous section). Note that an object may have some strictly local methods, and others, denoted GLOBAL, potentially globally available. Methods of global objects are restricted by the language to prevent local references, implicit via var parameters and explicit local pointers, from escaping their local address space via method parameters. However a local object pointer can always be mapped to a global one. In fact the GLOBAL type annotation is applicable to any type, it is e ectively applied recursively through types as far as pointers, eg:
hINTEGERi ) INTEGER hRECORD l:T ENDi ) RECORD l: GLOBALhTi END hPROCEDURE(x: T)i ) PROCEDURE(x: GLOBALhTi) hPOINTER TO Ti ) GLOBAL POINTER TO T
GLOBAL GLOBAL GLOBAL GLOBAL
Hence, the formal parameters of global methods have the GLOBAL type annotation implicitly applied to them. Thus any local object pointers become global ones. Note an Oberon style type test and guard may be used to turn a global object back to a local one if it is indeed local: VAR a: GLOBAL Accum; ... WITH a: Accum DO a.Add(5) ELSE a.GAdd(5) END
Note that the above \optimisation" to perform a local call in cases where the GO is indeed local is not required since method calls on GOs that happen to be local (or just on the same processor) are almost as ecient as local procedure calls. The method by which GO references are initially distributed between address spaces is unspeci ed in Mianjin. In Gardens the fork operation, which creates a task, can pass a single GO reference to the new task; this is sucient to bootstrap intertask communication.
A poll operation permits GO method invocations to occur, ie communication between address spaces. The poll operation is realised via a simple library function, Poll. For example a piece of code to utilise the sum abstraction: VAR o: Accum; ... NEW(o); o.sum := 0; o.count := 0; (* create NTasks tasks, pass them global references to o *) WHILE o.count
