interface to processes and to modules activated in a procedure-like mannner. (from now on .... the module denoted by X so that the operations "call" "activate", and "results" ..... ACM National Conference, June 1972~ pp 278-281. Walden ...
ON
UNIFYING
Bjarne
MODULE
INTERFACES
Stroustrup
The C o m p u t e r L a b o r a t o r y U n i v e r s i t y of C a m b r i d g e England
Abstract This p a p e r p r e s e n t s the o u t l i n e of a u n i f o r m i n t e r f a c e m e c h a n i s m for a c t i v a t i n g d i f f e r e n t kinds of modules, e.g. processes, monitors, and procedures. The usefulness of such an i n t e r f a c e in the d e s i g n of m o d u l e s and in the tuning of a s y s t e m is discussed. The o v e r h e a d s i n v o l v e d in u s i n g it are explained, t o g e t h e r w i t h some i m p l i c a t i o n s that its general use has on the s y s t e m structure.
Introduction The low price of h a r d w a r e has lead to the p r o l i f e r a t i o n of hardware a r c h i t e c t u r e s and in p a r t i c u l a r o p e n e d n e w p o s s i b i l i t i e s for u s i n g special h a r d w a r e , extra processors, extra f r o n t - e n d computers, and even c o m p l e t e i n d e p e n d e n t systems to enhance a basic s y s t e m by s p e e d i n g up s p e c i f i c functions or just to take some load off t h e main p a r t o f the system. The e x p l o i t a t i o n of such s e m i - d i s t r i b u t e d h a r d w a r e systems b y migration o f software m o d u l e s from the m a i n p a r t of the s y s t e m to s u p p o r t i n g h a r d w a r e is u s u a l l y greatly h a m p e r e d by t h e lack of a p r e c i s e d e f i n i t i o n of modules and their d e p e n d e n c i e s , so that the m o v e m e n t of a m o d u l e involves changes both to it and to all m o d u l e s w h i c h activate it directly. Even simple m u l t i p r o c e s s o r systems p r e s e n t p r o b l e m s of h o w to o r g a n i z e software to e x p l o i t the extra p a r a l l e l l i s m o f f e r e d b y the hardware, b e c a u s e it is d i f f i c u l t to vary the a m o u n t of p o t e n t i a l p a r a l l e l l i s m o f f e r e d b y the basic s o f t w a r e to suit all the p o s s i b l e numbers o f processors. In general one w o u l d e x p e c t that the d e c i s i o n of w h e t h e r a given module s h o u l d be i m p l e m e n t e d as a p r o c e s s or as a p r o c e d u r e w i l l d e p e n d on the h a r d w a r e c o n f i g u r a t i o n (and the job load). A u n i f o r m i n t e r f a c e to all modules p r o v i d e s a p o s s i b l i t y for v a r y i n g the i m p l e m e n t a t i o n of modules to suit a w i d e range of situations, e.g. a u n i f o r m interface to both local a n d remote p r o c e s s e s (i.e. p r o c e s s e s r u n n i n g on this m a c h i n e and p r o c e s s e s r u n n i n g on some other machine) allows one to w r i t e a n e t w o r k i n t e r f a c e p r o c e s s w h i c h runs u n c h a n g e d e i t h e r on the main c o m p u t e r or on a c l o s e l y a t t a c h e d "NIP" computer. A uniform i n t e r f a c e to p r o c e s s e s a n d to m o d u l e s a c t i v a t e d in a p r o c e d u r e - l i k e m a n n n e r (from n o w on simply c a l l e d procedures) allows one to tune a s y s t e m by p r o v i d ing m o r e or less p r o c e s s e s to suit the actual n u m b e r of processors, e.g. a d i r e c t o r y m a n a g e r m o d u l e i m p l e m e n t e d as a p r o c e d u r e on a u n i - p r o c e s s o r i n s t a l ation could be m a d e into a p r o c e s s after the i n s t a l l a t i o n has b e e n e n h a n c e d w i t h a s e c o n d processor.
90
It must be kept in mind throughout that the unification of the interfaces to the various kinds of modules is done to ease programming and to provide a facility for varying the calling aspects of their implementation w i t h o u t re-programming - not as a substitute for human understanding of the w a y the system runs. If modules in a system typically are very large and complex such a variability would be extremely hard to achieve at a reasonable low cost b e c a u s e of the consequent complexity of the calling conventions and mcdule specifications. Even if such an interface was defined the apparent variability would turn out to be useless as there would be relatively few such modules, and b e c a u s e large complex modules tend to rely on many resources being directly available so that they cannot easily be moved or changed. In the following it will therefore, be assumed that modules are "quite small" and "simple", i.e. of the type and magnitude found in systems based on the p r i n c i p l e of m i n i m u m privilege, type extension principle, data abstraction principle, etc.. This implies that a typical system module would be measured in h u n d r e d s of lines of (HHL) code, as opposed to tens or thousands, and would rarely access more than one "resource" at a time. The Cambridge
Programming
not)
CAP system
is a prime
example
of this.
Considerations
Modules which try to implement appear to be of a common form:
one single
B E G I N # pseudo Algol68 module # i n i t i a l i z a t i o n code; DO # to infinity # get arguments; CASE first argument IN # one possible
abstraction
convention
(and many which do
#
: o .
services
offered
ESAC; return
results OD END; # of module # Examples of modules conforming to this format are CAP p r o t e c t e d procedures, most CAP processes, Simu!a67 classes, types, monitors, etc. in languages influenced by Simula. Obviously the CASE statement can be removed and services be o f f e r e d as procedures where a language dependent interface is acceptable. Similarly the "get arguments" and "return results" can be made implicit. If the module is implemented as a process the "get arguments" statement expands to the systems normal inter-process communication "receive a message" code,
e.g.
91
W H I L E messages (input) X :: receive message; where
X is of some
standard
=
none
"message"
DO wait
event OD;
mode~
If on the other h a n d the module is i m p l e m e n t e d as a p r o c e d u r e a simpler scheme can be u s e d for a s s i g n i n g a value to X, b u t again the a s s i g n m e n t can only be m a d e i m p l i c i t at the cost of making the s y s t e m language dependent, and does exist at some lower level anyway. In n e a r l y all systems and languages the decision of how to call a m o d u l e (e.g. s h o u l d it be a p r o c e s s or a procedure) has to be taken at the time w h e r e the module is designed, b e c a u s e o f slight variations in the p r o g r a m to i m p l e m e n t the r e c e i v i n g end of the interface. A u n i f o r m s t r u c t u r e (dependent on the existence o f a u n i f o r m c a l l i n g mechanism) p o s t p o n e s this type of d e c i s i o n to compile, s y s t e m generation, link, or even call time for m o d u l e s which are not d i r e c t l y concerned with parallellism. This simplifies the initial w r i t i n g o f the module and makes it possible to change the module w i t h r e s p e c t to the c a l l i n g i n t e r f a c e w i t h o u t r e - w r i t i n g and maybe w i t h o u t r e - c o m p i l i n g it. A u n i f o r m w a y o f w r i t i n g m o d u l e s is n e c e s s a r y for a c h i e v i n g this d e g r e e of variability, but a u n i f o r m c a l l i n g i n t e r f a c e can be i m p l e m e n t e d w i t h o u t enforcing (or defining) such a s t a n d a r d module format. This implies that one can allow n o n - s t a n d a r d modules in the system, e.g. p r o c e s s e s w i t h m o r e than one r e q u e s t channel - w i t h o u t losing the o t h e r b e n e f i t s o f a u n i f o r m c a l l i n g interface. The u n i f o r m i n t e r f a c e i t s e l f w o u l d free the p r o g r a m m e r from the n e e d to know the i m p l e m e n t a t i o n of modules he wants to call in o r d e r to g e n e r a t e the correct calling sequences. If b e n e f i t s could c o n c e i v a b l y be o b t a i n e d f r o m parallel execution o f a called module, then the c a l l i n g module could be p r o grammed using a sequence of o p e r a t i o n s s p e c i f y i n g the d e s i r e d p a r a l l e l l i s m , e.g. activate
Y
:= results;
otherwise Y
(X, arguments);
# implies
a procedure-like := call
possible
activation
wait
for X to f i n i s h
w o u l d be used.
#
e.g.
(X, arguments);
It m u s t then be assumed t h a t t h e reference (pointer, descriptor, capability, or whatever) X contains enough i n f o r m a t i o n to i d e n t i f y the type of the m o d u l e d e n o t e d by X so that the operations "call" "activate", a n d "results" can be i n t e r p r e t e d so as to use the systems s t a n d a r d procedure, process, monitor, or r e m o t e p r o c e s s c a l l i n g conventions. The o v e r h e a d of this is q u i t e low. One s t a t e m e n t to find or isolate the type, plus one case s t a t e m e n t of t h e s i m p l e s t form to s e l e c t the s e t of c a l l i n g c o n v e n t i o n s that the p r o g r a m m e r w o u l d h a v e s p e c i f i e d d i r e c t l y in a traditional system. An e x t r a o v e r h e a d is i n c u r r e d w h e n p o s s i b l e p a r a l l e l e x e c u t i o n is s p e c i f i e d w h e r e the a c t i v a t e d m o d u l e is in fact a procedure, b u t this is again m a r g i n a l as it consists m a i n l y in p r o v i d i n g
92
storage for the results takes place~
while
the
"parallel"
execution
of t h e callers m o d u l e
As an a l t e r n a t i v e to a u n i f o r m calling i n t e r f a c e one can hide the actual i m p l e m e n t a t i o n of a m o d u l e b e h i n d a library routine, or as in the CAP system b u r y it b e h i n d a p r o t e c t e d p r o c e d u r e acting as an interface or m a y b e e v e n e n c a p s u l a t i n g and p r o t e c t i n g unknown p r o c e s s e s and/or procedures. This standard t e c h n i q u e gives some of the c o n v e n i e n c e of a u n i f o r m i n t e r f a c e at the cost of an extra layer o f software, but it is not general unless there is some w a z o f e n f o r c i n g that all a c t i v a t i o n s of a module "hidden" b e h i n d an i n t e r f a c e p r o c e d u r e in fact go t h r o u g h that procedure. If this is not the case anyone w a n t i n g to change th i m p l e m e n t a t i o n of the m o d u l e w i l l still h a v e to m a k e changes to (an unknown n ~ n b e r of) user p r o g r a m s - some of w h i c h he m i g h t not have the p e r m i s s i o n to change. Also, if the e n c a p s u l a t i o n is not u s e d for all modules a progran~ner will still have to find out w h e t h e r (and if so how) his p a r t i c u l a r module is e n c a p s u l a t e d . If the use o f i n t e r f a c e p r o c e d u r e s is enforced at compile time the s y s t e m is likely to b e c o m e d e p e n d e n t on a single language, and if the use is e n f o r c e d d y n a m i c a l l y by some form of c a p a b i l i t y m e c h a n i s m the interface becomes i n t o l e r a b l y slow - though this o v e r h e a d is quite a c c e p t a b l e in special cases and has b e e n used w i t h g r e a t success in the CAP system.
A Set of I n t e r M o d u l e C o m m u n i c a t i o n
Primitives
There are of course numerous p o s s i b l e sets of inter m o d u l e c o m m u n i c a tion p r i m i t i v e s w h i c h fulfil the general d e s i g n aims. The set p r e s e n t e d b e l o w s h o u l d be s e e n as o n l y one that suited a p a r t i c u l a r p r u p o s e and environment. The s c h e m e is b a s e d on p a s s i n g of fixed s i z e d a r g u m e n t blocks (though the p r i m i t i v e o p e r a t i o n s c o u l d be d e f i n e d using o t h e r data structures). An a r g u m e n t b l o c k is used b o t h for p a s s i n g a r g u m e n t s and r e t u r n i n g results. It contains a few words w i t h o u t fixed i n t e r p r e t a t i o n , b u t normally i n t e r p r e t e d as integers, plus a few words i n t e r p r e t e d as c a p a b i l i t i e s (references, d e s c r i p tors, pointers). In requests the "first a r g u m e n t " is c o n v e n t i o n a l l y i n t e r p r e t e d as an i n t e g e r s p e c i f y i n g the s e r v i c e required, a n d in results it is i n t e r p r e t e d as a return code. Zero i n d i c a t e s n o r m a l return. The "last argument" is a c a p a b i l i t y i n t e r p r e t e d as a m o d u l e return link. This implies that a m o d u l e cann o t (in general) be c a l l e d r e c u r s i v e l y , t h a t t h e s y s t e m can be d e a d l o c k e d b y r e c u r s i o n o r m u t u a l recursion, and that a module can "return" to a module d i f f e r e n t from its caller. If the "last a r g u m e n t " is not s p e c i f i e d no reply is expected, no results can be returned, a n d the m o d u l e can w a i t for a new request. The p r i m i t i v e
operations
are-
ACTIVATE
(X,A)
will run the m o d u l e i d e n t i f i e d b y X w i t h the a r g u m e n t b l o c k i d e n t i f i e d b y A. If p o s s i b l e X s h o u l d run in p a r a l l e l w i t h the c a l l i n g module.
93
Y:= G E T A R G U M E N T S
(C)
will w a i t for some A C T I V A T E or CALL of the c u r r e n t m o d u l e to occurl and then m a k e Y denote the a r g u m e n t b l o c k sent. The a r g u m e n t C d e n o t e s a n o t i o n a l c o m m u n i c a t i o n channel. It can take two values "request" and "reply"° This m a k e s it p o s s i b l e to s e p a r a t e n e w r e q u r e s t s f r o m replies° This is a m i n i m a l f a c i l i t y w h i c h can be e x p a n d e d to allow more c o m p l i c a t e d s t r a t e g i e s for r u n n i n g m o d u l e s in parallel°
is e q u i v a l e n t
Y:= C A L L
(X,A)
ACTIVATE
(X,A) ; Y
to
so it is not l o g i c a l l y a p r i m i t i v e operation.
necessary,
but
:= G E T A R G U M E N T S
(reply)
can be i m p l e m e n t e d m o r e e f f i c i e n t l y
as
T h e r e is o b v i o u s l y a n e e d for p r e c i s e s p e c i f i c a t i o n o f the s i z e o f a r g u m e n t blocks, t h e i r m a n a g e m e n t , the u n d e r l y i n g m e s s a g e s y s t e m for local and remote p r o c e s s e s , how the "module r e t u r n link" finds its w a y i n t o t h e "last argument", w h a t r e f e r e n c e s are and h o w they are translated, w h a t t h e c o m m u n i c a t i o n c h a n n e l s u s e d b y G E T A R G U M E N T are, etc., b u t this is b e y o n d the scope of this paper. It can be s e e n though that a m o d u l e can be w r i t t e n w i t h o u t k n o w l e d g e o f the imp l e m e n t a t i o n of its c a l l i n g i n t e r f a c e (which t h e r e f o r e is v a r i a b l e ) , and that it can a c t i v a t e o t h e r m o d u l e s s e r i a l l y or in p a r a l l e l w i t h o u t k n o w l e d g e of t h e i r implementation: BEGIN # module # initialize; ~ DO arg := G E T A R G U M E N T S (request) ; CASE first a r g u m e n t O F arg IN •
°
one service: •
o
res := C A L L (A,arg2) ; IF first a r g u m e n t O F res ~= 0 T H E N e r r o r o
another •
FI;
.
service :
•
ACTIVATE
(B,arg2) ;
res := G E T A R ~ J M E N T S (reply) ; IF first a r g u m e n t OF res I= O T H E N e r r o r FI; •
J
ESAC
ACTIVATE
(last a r g u m e n t O F arg,
OD END;
# of m o d u l e
#
94
results)
One final p r o b l e m remains though. A process or a m o n i t o r module can access r e s o u r c e s used by it alone w i t h o u t any f o r m of interlocking, as an i n t e r l o c k m e c h a n i s m is implicit in its implementation, b u t a m o d u l e i m p l e m ented as a p r o c e d u r e with instances in many p r o c e s s e s m u s t use an interlock. The s i m p l e s t solution to this is to use a simple i n t e r l o c k m e c h a n i s m on e a c h access to such resources in all cases, k n o w i n g that the o v e r h e a d i n v o l v e d in finding it l o c k e d will n e v e r be paid in the process and m o n i t o r cases. This will ensure that the p o t e n t i a l advantage in using p r o c e d u r e s in the case w h e r e the critical s e c t i o n is short r e l a t i v e to the w h o l e module will be obtained. If the s y s t e m does not contain an interlock m e c h a n i s m w h i c h reduces to a single test i n s t r u c t i o n in the u n l o c k e d case, such a m e c h a n i s m can e a s i l y be p r o v i d e d to ensure that the (expensive) s y s t e m interlock m e c h a n i s m w i l l o n l y be u s e d w h e n necessary.
Implications
for S y s t e m S t r u c t u r e
There are three m a j o r c o n s t r a i n t s w h i c h the i n t e r c h a n g e a b i l i t y of p r o c edures, processes, and remote p r o c e s s e s imposes on the general structure o f the system. There can be no i m p l i c i t p a s s i n g of i n f o r m a t i o n in calls between p r o cedures in the same process. This is a r g u a b l y a good thing anyway, b u t for e f f i c i e n c y reasons tricks like not saving the registers in this case and so a c h i e v i n g a cheap w a y o f p a s s i n g integers is not uncommon. The global context for a job cannot be p r o v i d e d in the data s t r u c t u r e d e f i n i n g the c u r r e n t p r o c e s s (process base, state word) as the current p r o c e s s is quite likely to consist of only the module itself. Global p a r a m e t e r s like the current input s t r e a m and the user identifier m u s t e i t h e r b e b a n n e d and s u b s t i t u t e d b y explicit p a s s i n g or be p r o v i d e d as a special "current job" module or data structure. This has the e f f e c t of m a k i n g the p r o c e s s e s slightly cheaper to use, and also e a s i n g the i m p l e m e n t a t i o n of s c h e d u l i n g and store m a n a g e m e n t a l g o r i t h m s b a s e d on jobs rather than processes. Error codes m u s t always be r e t u r n e d to the c a l l i n g m o d u l e unless something else is s p e c i f i c a l l y requested. This is e s s e n t i a l in t h e case w h e r e the c a l l e d m o d u l e is i m p l e m e n t e d as a remote process a n d t h e r e f o r e s u s c e p t i b l e to soft errors .
Reconfiguration
of S o f t w a r e
Given a u n i f o r m inter m e d u l e c o m m u n i c a t i o n s y s t e m one can e x p e r i m e n t with the i m p l e m e n t a t i o n cf modules. This allows one to o b t a i n data r e l a t e d to design d e c i s i o n s in the area of p a r a l l e l l i s m / s e r i a l i z a t i o n , and to tune a s y s t e m to a specific h a r d w a r e c o n f i g u r a t i o n and/or work load. In this w a y the choice of i m p l e m e n t a t i o n o f a m o d u l e c a n be seen as the first and m o s t p o w e r ful of the s y s t e m short t e r m s c h e d u l i n g p a r a m e t e r s . As a m a t t e r o f fact if one is w i l l i n g to a c c e p t a p r o c e s s like format for all m o d u l e s most o f the effects of using the i n t e r f a c e for s p e c i f y i n g m o d u l e s as p r o c e s s e s and monitors can b e o b t a i n e d by u s i n g a flexible s c h e d u l i n g a l g o r i t h m w i t h extreme parameters.
95
Given the variability and consequently the possibility for data collection through experimentation there are three possible ways of exploiting the interface as a tuning tool: i: simple experimentation and adjustments of the implementation by a system managere who can specify the implementation of modules at system generation time in much the same way as he adjusts other scheduling parameters° Changes in the system (in particular in connection with remote processes) will be infrequent and whollely dependent on human decisions. 2: Adaptive algorithms can be introduced into the system controlling the implementation of all modules. For this to be possible the implementation of the uniform interface must be capable of handling changes on the fly. This in itself is costly (but possible), and a suitable algorithm w o u l d not be easy to find. This has been attempted with distributed systems . 3: It is conceivable that an algorithm could be found which given a specification of the hardware, the module interdependencies, and a "typical" work load could compute a suitable set of implementations for the modules. This w o u l d be ideal, but even specifying the inputs w o u l d be extremely difficult. An additional p r o b l e m with reconfiguration must be mentioned. It is possible to introduce a deadlock into the system by taking a module which acquires two resources in a time dependent manner and has previously been implemented as a process and making it into a procedure. Such problems must be handled by human intelligence - the u n i f o r m interface is not designed as a substitute for human knowledge of the system implementation, only as an aid in the design phase and for subsequent tuning.
Possible
Implementation
Methods
The uniform inter module coFmmunication m e c h a n i s m is an interface to the usual calling mechanisms rather than a separate new mechanism. This implies that the implementation problems lie in avoidance of overhead rather than in actual building of new facilities. There are basically ~wo ways of implementing it; One can implement the communication primitives in something which can recognize the various types of modules and then p e r f o r m the appropriate actions. Because of its complexity this "some~ling" must be a high level facility - a module or a set of modules. This implies that one will incur a large fixed overhead in all cases and for that reason end up finding that the use of any kind of module will have a cost of the same order of magnitude as a process. The alternative is to allow the simplest inter module communication primitive - e.g. a microprogrammed instruction for procedure like call of a module (i.e. no process switch or interlocking implied) - to escape to more complicated and therefore slower mechanism when the primitive is passed a reference it cannot interpret. If this technique is used in all cases the extra overhead will be nil in the simplest case and one escape action in the next simplest case etc. As an example one can take a CAP like system. The micropregram can handle the simple (protected) procedure call. A module in the
96
callers own process (known to the microprogram) can handle calls to other p r o c e s s e s via the m e s s a g e system, and calls to remote processes will have to be r e d i r e c t e d to a network interface process known to the simple message system. This m e t h o d minimizes the overhead in all cases.
Conclusion It is possible to write most modules in a system w i t h o u t c o n s i d e r i n g their p o s s i b l e i m p l e m e n t a t i o n as a procedure, process, monitor, or whatever. In d o i n g so one achieves a powerful tool for tm~ing the software to specific h a r d w a r e configurations and jobloads w i t h o u t noticeable overheads, q~e usefulness of this technique appears to be limited to systems in which modules are fairly small, context swtiching not too expensive, and w h e r e i n f o r m a t i o n is p a s s e d explicitly rather than relying on shared data.
F u t u r e Work An inter module communication system of the class d e s c r i b e d in this p a p e r has b e e n used as the interface between modules in an o p e r a t i n g s y s t e m b a s e d on the CAPs o p e r a t i n g system. This system is w r i t t e n in Simula67 and is running s y n t h e t i c jobloads on a variety of (simulated) hardware configurations. It is u s e d for p r o v i d i n g data for a project to evaluate the use of procedures, m o n i t o r s , and processes, and to explore the possibilities for using remote p r o c e s s e s to e x p l o i t s e m i - d i s t r i b u t e d hardware configurations. Similar schemes for u n i f y i n g process and procedure calls are being m i c r o p r o g r a m m e d for the CAP computer.
Acknowledgements This w o r k was done while holding a research fellowship from the Danish Natural Science R e s e a r c h Council. I am grateful te Roger M N e e d h a m for c o n s t r u c t i v e comments on an earlier version of this paper.
References
Casey, L. and Shelness, N. A Domain S t r u c t u r e for D i s t r i b u t e d Computer Systems A C M - O S R vol ii no 5 (Nov.77) pp 101-108
Farber, D.J. et al. The D i s t r u b t e d C o m p u t i n g S y s t e m 7th annual IEEE Comp. Conf.(1973)
Needham, R.M. and Walker, R.D.H. The Cambridge CAP Computer a~d its P r o t e c t i o n A C M - O S R vol Ii no 5 (Nov. 77) p p i-iO
Needham, R.M. The C A P Project - An Interim E v a l u a t i o n A C M - O S R vol ii no 5 (Nov.77) p p 17-22
97
System
Metcalfe, Robert M Strategies for Operating Systems in Computer Networks Proc. ACM National Conference, June 1972~ pp 278-281
Walden, David C. A System for Interprocess Communication in a Resource Sharing Computer Network CACM vol 15 no 4 (April 72) pp 221-230
98
