RPC is executed by the calling process in the context of the server (after ... ranges needed to support such a system and it is only the emergence of cheap 64 bit.
Angel: a proposed multiprocessor operating system kernel
(Extended Abstract) T.Wilkinson, T.Stiemerling and P.Osmon Computer Science Department, & City University, Northampton Square, London EC1V 0HB, UK.
A.Saulsbury and P.Kelly Department of Computing, Imperial College, 180 Queens Gate, London SW7 2BZ, UK.
1 Introduction We describe an experimental multiprocessor operating system called Angel. The design of Angel builds on experience gained from constructing the Meshix operating system [1] and from the implementation of a distributed shared memory server on this system [2]. The aim is for Angel to support a single uniform shared memory address space on a multiprocessor machine which contains both shared and distributed memory. Such a machine is seen to be a hierarchical system containing multiprocessor clusters using physically shared memory, which are then loosely coupled using a multi-path network. A number of current operating systems have a micro-kernel implementation and support lightweight threads, examples being Mach [3] and Chorus [4]. In these systems the term process can be used to de ne a domain of protection, where inter-process communication (IPC) occurs through message-passing, while a process may be composed of many threads sharing the same address space and communicating through shared memory. A crucial issue here has been improving the eciency of IPC, especially for processes on the same node. In distributed memory systems the execution of the threads in a process is restricted to the same node. This restriction, and the desire to provide a shared memory programming environment on such systems (in preference to using message-passing), has led to various implementations of distributed shared memory (DSM) [5]. We propose to integrate DSM mechanisms into the Angel kernel so that the entire system's address space is uni ed under a single DSM region which handles the caching of code, migration of data and coherency of distributed data structures. The objective is also to hide the seams where bus-based snooping, directory based multi-casting and DSM are used, and to do so in a secure fashion while retaining the performance expectations of the shared memory programming model.
2 Improving OS eciency Although a micro-kernel implementation increases the simplicity and exibility of the operating system, performance is impacted by the increased parameter/result copying, virtual memory re-mapping and context-switching necessary for the execution of basic system functions. An example is remote procedure call (RPC), currently the dominant abstraction for IPC.
In a monolithic Unix system, a local RPC is executed in kernel mode in the context of the calling process, and parameter copying may be required if the calling process is re-scheduled. In a message-passing kernel, like Meshix a naive RPC implementation requires excessive parameter copying, and Meshix therefore re-maps parameters into a shared segment (both Mach and Chorus optimise this by copying small parameters). In lightweight RPC (LRPC) [6], parameters are passed in a shared segment, and the RPC is executed by the calling process in the context of the server (after parameter validation). With a single address space we can use a simpli ed LRPC mechanism, because it is made easy and natural for processes to share access to common data regions. We also have a way of unifying local RPC with non-local (message-passing) RPC, using the DSM to migrate calling processes to the appropriate node. By clarifying and unifying these mechanisms we regain some of the simplicity and performance of the conventional Unix system call mechanism. We have replaced a client-server interaction based on message passing by one based on a thread moving from client to server and back again, pausing only for security validation. Unlike the Unix case, we can have an arbitrarily-complex protection structure; the cost is copying the parameters once each time a protection boundary is crossed. We have also gained the freedom to migrate the server at run-time in a completely transparent fashion. The remaining cost of cross-domain calls arises from the need to change the memory map, which involves cache ushing or invalidation. Some of the cache re lling cost is attributable to excess ushing/invalidation, and this can be avoided if the address space is alias free. Similarly, ushing and invalidation can be avoided when moving from one domain to another if the new domain has a superset of the rights of the old one.
3 An overview of Angel Angel aims to provide a simple parallel programming model regardless of the physical machine, or machines, on which it resides. It does this by providing four basic building blocks; objects, capabilites, processes and sychronisation primitives. Angel like Multics [7], only supports a single address space. In Angel this is the only namespace and all notions of identity, protection and synchronisation are made based on addresses within this space. 32 Bit processors cannot provide the address ranges needed to support such a system and it is only the emergence of cheap 64 bit processors which make this viewpoint feasible. An object is a protected, xed sized, xed position area of memory within the Angel namespace. In a small virtual address space this rigid structure would be unacceptable, but with 64 bit addresses, the declaration of gigabyte objects just in case the space is needed makes little impact on the remaining space. Object are always persistent, and will be garbage collected when there are no futher references to them (although it is possible to delete them explicitly). Object references also provide the means of protecting objects and are called capabilities. Capabilities are entities which provide access to designated objects and describe what operations may be performed upon them. It can be compared to a lename providing access to a le. Capabilities in Angel do not have an enforced structure, only a convention. They are handled by the services which create and destroy objects and so could provide capabilities in di erent ways if desired. Capabilties may be stored in any object and are represented only be an address, the address of the service it is provided by. It is the position in the namespace which is
important and they may only be moved and copied by the issuing service. All the related information; object start address, length, access rights and soforth; is maintained in the service's protected data space. Consequentally, tampering is impossible; if a reference is made for a capability at an unknown address it is obviously a forgery. A process is the locus of control. It may be created by another process though the use of spawn, and may be destroyed either voluntarily or by another (rights permitting). Physically, a process consists of a control object which holds all related information. The majority of this consists of the objects the process has chosen to access by resolving possessed capabilites with the relevant services, so mapping the objects into its view of the namespace. It should be noted that possession of a capability by a process does not necessarily allow it to be resolved by that process. Synchronisation between processes is provide by a sleep/wakeup mechanism. The rendezvous points are addresses within accessable objects, allowing synchronisation to take place between physically remote processes as easily as between local processes. Unlike Unix sleep/wakeup, a wakeup will persist until it is collected by an assoicated sleep. This avoids synchronisation race problems in shared memory machines.
4 Structure of Angel
Angel is designed as a layer micro-kernel; a set of services constructed in layers. These layers are described below. The rst two incorporate the trusted machine speci c and device speci c services; the third provides the Angel environment; and the last the high level services and applications. Local services reside on every machine comprised in an Angel system. The services include process context scheduling, exception dispatch, and virtual-to-physical memory translation. The implementation of these services is machine speci c. Device drivers will provide a standard interface to various peripherals including physical memory, disk storage and ethernet. These services are used by the distributed management services for object storage and communication with other machines. Distributed management services are responsible for unifying a number of machines into a single Angel system. For objects and synchronization, it does this though the use of DSM techniques. For processes, it uses RPC to export control to the necessary machine. System services provide Angel programs with services which are useful to applications programs. These not only include libraries of routines but also protocol services, lesystems and object/capability managers.
5 Applications under Angel
The de nition of Angel supports, but does not include, higher level operating system functions such as system call libraries, shells and lesystems. These must be provided by seperate server programs. This allows other existing operating system interfaces to be emulated, in much the same way as Mach and Chorus support Unix1. Appropriate applications, such as a database managament system, may also be ported directly on to the bare kernel. The object/capability structure of Angel leads to a natural implementation of a lesystem, in which les are permanent objects and directory servers bind these objects 1The
Angel
single address space poses some problems for implementing
though.
Unix
semantics correctly on
to symbolic lenames. Once a le has been accessed by a directory server, it is mapped into a process's address space and can be accessed directly.
6 Conclusion, future work and acknowledgements
Experience with Meshix has led us to propose Angel, a multiprocessor operating system kernel. With the objectives of providing a clean programming model, and providing structural independence from details of how data sharing is implemented, we have motivated the use of a single coherent shared address space as a fundamental structure to support both parallel applications programs and system services in Angel. We have described an operating system with desirable properites, but have not yet veri ed that this architecture can be implemented eciently enough for the programming model presented to be acceptable. A number of implementation stages are planned along with concurrent machine development. The rst stage implementation of Angel will be on a uniprocessor MIPS R4000 machine, the second stage migrating to a message-passing cluster and then nally to a multiple shared-memory cluster machine as they become available. Thanks to M.Cripps (Imperial College), P.Winterbottom (AT&T Bell Labs), A.Whitcroft and A.Gull for their help and encouragement. Financial support was provided by City University and the UK SERC.
References [1] P. Winterbottom and T. Wilkinson, \MESHIX: a UNIX like operating system for distributed machines," in UKUUG Summer Conference Proceedings, pp. 237{246, July 1990. [2] A. Saulsbury and T. Stiemerling, \A DVSM server for Meshix," Tech. Rep. (forthcoming), City University Computer Science Department, October 1991. [3] R. Rashid, \From RIG to Accent to Mach: the evolution of a network operating system," in Proceedings of the ACM/IEEE Computer Society Fall Joint Conference, November 1986. [4] M. Rozier and L. Martins, Distributed Operating Systems: Theory and Practice, Nato ASI Series, vol. F28, ch. The Chorus distributed operating system: some design issues, pp. 261{287. Springer Verlag, 1987. [5] B. Nitzberg and V. Lo, \Distributed shared memory: A survey of issues and algorithms," IEEE Computer, vol. 24, August 1991. [6] B. Bershad, T. Anderson, E. Lazowska, and H. Levy, \Lightweight remote procedure call," ACM Operating Systems Review, vol. 23, pp. 102{113, December 1989. [7] E. Organick, 1971.
. The MIT Press,
The MULTICS system: an examination of it's structure
