ULTRIX client Systems. The prototype HDS has included at various times a. ' VMS, ULTRIX,. VAX, and PDP-11 are trademarks of Digital Equipment Corporation, ...
A Programmable Interface Language for Heterogeneous Distributed Systems JOSEPH R. FALCONE Digital Equipment Corporation
The 1980s have witnessed the emergence of a new architecture for computing based on networks of personal computer workstations. The performance requirements of such systems of workstations places a strain on traditional approaches to network architecture. The integration of diverse systems into this environment introduces functional compatibility issues that are not present in homogeneous networks. Effective prescriptions for functional compatibility, therefore, must go beyond the communication paradigms used in present distributed systems, such as remote procedure calls. This paper proposes a distributed system architecture in which communication follows a programming paradigm. In this architecture a programming language provides remote service interfaces for the heterogeneous distributed system environment. This language is a flexible and efficient medium for implementing service function protocols. In essence, clients and servers communicate by programming one another. CR Categories and Subject Descriptors: C.2.4 [Computer-Communication Networks]: Distributed Systems--distributed appZications; network operating systems; D.3.2 [Programming Languages]: Language Classifications-applicative languages; extensible languages; D.4.7 [Operating Systems]: Organization and Design-distributed systems General Terms: Design, Experimentation,
Languages, Standardization
Additional Key Words and Phrases: Distributed systems, heterogeneous remote function evaluation, remote procedure call
systems, local area networks,
1. INTRODUCTION The arrival of personal computer workstations in the modern office has created new problems for system engineers. Until recently, these personal computers did not offer networking capability, forcing each workstation to maintain its own printer and storage resources. Now the installation of high-bandwidth communication systems is progressing rapidly, but the software supplied with these networks usually provides little more than file transfer and mail delivery facilities. Workstation users demand much more than these features; hence the goal of most research efforts in this field has been the development of a true distributed system. A true distributed system migrates common operating system services from the personal computer workstations to special nodes known as servers. Author’s address: Digital Equipment Corporation, 333 South Street, (SHRl-3/E29), Shrewsbury, MA 01545-4132. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. 0 1987 ACM 0734-2071/87/1100-0330 $01.50 ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987, Pages 330-351.
A Programmable Interface Language
l
331
However, the design of such a system is particularly complex because of the tight coupling between applications and the cheap, local resources in most operating systems. When one moves these resources out to a server on the network, the performance of the resulting distributed system can suffer. Most existing distributed systems insist on a homogeneous population. We could not assume this arrangement since Digital Equipment markets several different operating systems (VMS, ULTRIX, MS-DOS) implemented on several machine architectures (VAX, PDP-11, Intel 8086).’ ULTRIX is Digital’s proprietary version of UNIX [ 19].2 Despite the fact that these systems and architectures share many features, they do not naturally integrate into a unified distributed system. In 1983, our project at the Eastern Research Laboratory began to develop an architecture to link Digital’s diverse operating systems into a highperformance heterogeneous distributed system (HDS) supporting many distinct applications. Developments in heterogeneous distributed systems have been slow in coming since implementation of the watershed ARPANET RSEXEC [21]. Indeed, in that article the implementors state that “[tlhere are, of course, limitations to how tightly Hosts with fundamentally different operating systems can be coupled.” Although an HDS implies a software and hardware diversity that is at odds with a tight coupling, there can be homogeneous points of compatibility across the constituent systems based on common views of resources. The nature of these points of compatibility determines the extent of resource sharing in the HDS. The most common choice for a point of compatibility is the service interface of a particular operating sysetem such as UNIX, although this makes it more difficult for other operating systems to join the HDS [4,22]. A similar approach is to develop custom operating systems geared for cooperation [ 171. One can also move the point of compatibility from the operating system into the transaction protocol for a particular application, such as an airline reservation system [9]. However, since one application dominates the distributed system, it is very difficult to support other applications efficiently. Our objective was to integrate a truly diverse collection of commercial hardware and software into a distributed system with shared operating system resources supporting many distinct applications. This paper describes the architectural glue that binds these diverse elements together into one unified system. Our prototype HDS follows a client/server model in which a number of centralized services are available to nodes on an Ethernet [5, 151 local area network. These services consist of those normally provided in a time-sharing system, including file storage, printer queues, name resolution, message broadcast, and error logging. The HDS uses its own services extensively; for example, a client uses the name service to find a particular file server. The server nodes offering these services are VAX/VMS and VAXELN systems in our prototype, but our architecture conceals this from the users of the VMS, VAXELN, and ULTRIX client Systems. The prototype HDS has included at various times a ’ VMS, ULTRIX, VAX, and PDP-11 are trademarks of Digital Equipment Corporation, MS-DOS a trademark of Microsoft Corporation, and Intel 8086 is a trademark of Intel Corporation. ’ UNIX is a trademark of AT&T Bell Laboratories. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
is
1987.
332
-
Joseph R. Falcone
VAX-11/780, two VAX-11/750s, four VAX-11/730s, MicroVAX II systems for clients and servers.3
and ten MicroVAX
I and
2. OVERVIEW For a distributed system to be effective, its diverse components must coordinate their efforts productively. The first step toward this coordination is the establishment of a communication mechanism among the participant systems. At the lower levels in the International Organization for Standardization (ISO) open systems interconnection model [25], we can work with a variety of network media and transports for the transmission of binary data between machines. At the higher levels, where the binary data assume meaning, all parties on the network must adopt the same conventions in order to maintain the semantic integrity of the distributed system. In particular, the clients and servers must select a common data representation and a common interface paradigm. Although many communication mechanisms address the issue of heterogeneous data compatibility [3,7, 231, few have attempted to resolve differences among service interfaces for operating system resources removed to HDS servers.
2.1 Service Interface Design The interface to a distributed system service is typically a collection of functions. In early distributed systems, these functions were just the native service functions on the server node. For a homogeneous population, this interface can provide the basis for a distributed operating system environment, such as Digital’s VAXcluster architecture [1O].4 However, this approach is inadequate for an HDS architecture because it requires all clients to comply with a single operating system standard. If a client needed features not in that standard, it would have to do without them. At the other extreme is an approach that allows each class of clients to determine its own interface to HDS services. Although this would satisfy the parochial needs of each class of clients, each server would have to support many different interfaces. Since this places unreasonable demands on servers, some systems dedicate servers to particular classes of clients. However, that approach precludes extensive sharing of resources by erecting the very same barriers between systems that communication networks were supposed to eliminate. One way to bridge distinctions among systems is to define a service interface as the collection of functions that constitute the least common denominator across all classes of clients. In this way, the interface consists of a basic function vocabulary for the service shared and understood by all clients. Some of these functions would be very simple, and a single client system service operation could translate into the invocation of many of these primitives across the network. Unfortunately, the communication overhead for exchanging messages to call these functions could well dwarf the execution time of the functions themselves. It is possible to tune such an interface for a particular class of clients by collecting 3VAXELN, VAX-11/780, VAX-11/750, VAX-11/730, MicroVAX marks of Digital Equipment Corporation. ’ VAXcluster is a trademark of Digital Equipment Corporation. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
I, and MicroVAX
II are trade-
A Programmable Interface Language
FILE
HLL-rX Compiler
HOST
l
333
B HLL-‘Y Compiler
+ A
Fig. 1.
Achieving
program portability
with a programming
language.
certain frequently used call sequences into new functions and thus exchange fewer messages, as is done in Internet Locus for UNIX clients [20]. However, it is neither practical nor feasible to track down every conceivable case for such special treatment in an environment where clients have truly diverse demands. As a result, it became clear that a new interface technique was needed to efficiently offer services in an HDS environment. 2.2 A Programming
Paradigm for Network Communication
Programming languages are the traditional human interface to computer services. Programs written in popular standard languages such as FORTRAN and COBOL are portable across diverse computer systems. In Figure 1, a programmer writes a single application in a standard high-level language that will compile and run on either Host A or Host B. This is possible because standard languages have a relatively small vocabulary with invariant semantics across different implementations. Indeed, the power of such languages lies in their enormous potential for implementing elaborate algorithms from terms in the vocabulary. The portability of computer programs suggests an analogous treatment for network communication. Consider an environment in which the means of communication are expressions sent from one computer (the generator) to another (the euahutor). In this way, the computers communicate in a network. FORTRAN, a linguistic bridge that eliminates system dependencies by allowing a client to specify in detail the functional behavior it expects from the server. In other words, the clients and servers communicate by programming one another. There already exist special purpose languages for the programming of print services to suit a client’s particular needs [l, 241, but we wanted to provide a general communication framework to encompass most services in a distributed system. Figure 2 shows such a mechanism with both Client A and Client B generating request expressions for a server system to evaluate. Note that the machine primitives of Figure 1 are replaced with the higher level abstraction of system calls in Figure 2. For efficiency, an HDS must limit itself to remote operations of a particular granularity, in this case, system calls rather than machine instructions. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
334
l
Joseph FL Falcone
SEKVER
Fig. 2. Heterogeneous systems using programming language expressions to request services.
Fundamental to this scheme is the development of a suitable base language, extensible to encompass all of the functions necessary for a full-service HDS. For our prototype, we designed the Network Command Language (NCLpronounced nickel) to fill this role. NCL represents the confluence of our distributed system design philosophy and the practical implementation demands of our prototype heterogeneous distributed system. The NCL programming model is symmetrical and not restricted to client/ server network architectures. Both client and server systems can use NCL to send an expression specifying the precise algorithm for an operation. Because the recipient systems evaluate the NCL expressions, a client or server can vary the programming dynamically to suit changing requirements within the HDS. In this way, a server can perform all of the operations necessary for a collection of heterogeneous clients without the prior definition of linkages or interface modules for every combination of primitives. As a result, the ultimate point of compatibility for the distributed system shifts to the NCL syntax, libraries, and canonical data formats. 2.3 Conceptual
Foundation
The roots of the Network Command Language are in the concepts embodied by LISP 1.5 [14]. The function and list orientation of the LISP model supports communication by programming very well. LISP is a good target language for the efficient automatic generation of expressions on client and server systems. The compact physical encodings of list structures reduce the memory overhead of S-expressions to an acceptable level. S-expressions have straightforward representations so systems can efficiently traverse and evaluate them. The adoption of the LISP model extends to the manner of communication between client and server. Whenever a process needs to communicate with a server, it establishes a session with the server, that is, it logs into the server by passing appropriate credentials (the level of security provided is an issue independent of NCL). If the process is authorized to use the service, then it enters what resembles an interactive LISP read-eval-print loop with the server evaluator, and the server connects back to the process the same way. The process and server exchange NCL expressions representing requests and responses over this session as if each were a user typing to an interactive evaluator. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language
-
335
Both LISP and NCL maintain little distinction between functions and data, so a function can go in place of any data element in an argument list. The traditional remote procedure call (RPC) model allows only one function per message [2,X3]. The LISP influence on NCL extends this model such that every parameter of each function
can itself be a function
that is evaluated remotely. So
instead of a remote procedure call, NCL supports the more general concept of remote function evaluation (RFE). Allowing functions as arguments also supports both composition and sequencing of functions, since any function can act as a control structure. Multifunction NCL messages save the overhead of multiple RPC invocations while reducing the complexity of servers on the network. Nonetheless, a system can efficiently use RFE with a single NCL function as a traditional remote procedure call. The true power of NCL lies in its function orientation. Instead of providing an extensive and complete programming language for all applications, NCL supports a small library of primitive functions that are the tools for constructing expressions with additional service-specific libraries. A system generates arbitrary expressions from the composition and sequencing of functions in these libraries. This approach permits clients to program HDS servers to mimic the functional behavior of their native operating systems. 2.4 Applications
For convenient access to server resources, our prototype utilizes existing interfaces to operating system services as the client window onto the distributed system. In so doing, any client system in the distributed environment can transparently accessserver resources by just making a system call. Using systemdependent criteria, the client executive identifies this system call as a request to a remote resource on a particular server in the HDS. For example, in some systems a particular file pathname prefix may indicate which server to deal with. Then the client executive presents the system call to its NCL generator for this service. Each client NCL generator features an operating-system-specific front end that interprets the information in the system call and a generic back end that produces a standard NCL expression to perform the desired operation on a server. The client sends the expression to the server who evaluates it using standard and service-specific NCL function libraries. The server generates a response expression as a by-product of the evaluation for passing the results back. The client receives the response expression and passes it to an NCL evaluator for disposition. In the VMS file system [8], the system call to access a file can include any or all of 5 modifiers (create, open, delete, extend, and truncate) and up to 31 attribute inquiries. Applications use an overwhelming number of combinations of these operations, so that it is not possible to boil the options down to a manageable number. This multioperation system call amortizes the expensive protection boundary crossing to the VMS file system executive. However, this poses the following dilemma for the designer of an HDS: How do you satisfy the VMS need for multiple operations initiated by an individual system call? One could implement this multioperation system call on an HDS file service either as a single complex procedure especially for VMS or as a collection ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
336
.
Joseph R. Falcone
of generic functions (access, create, open, delete, extend, truncate, and readattribute) shareable by all operating systems. The former case potentially results in a separate interface and implementation for each operating system. In the event that this complex procedure supersets the functionality desired by all heterogeneous clients, such that separate implementations are not necessary, this approach still suffers from having to test many option flags on each call regardless of whether or not the client would ever use them. Using traditional RPC, the latter case supports heterogeneity at the expense of multiple remote procedure calls for each system call. In our prototype, an NCL generator translates such complex system calls into an expression consisting of a few functions and variables. The VMS client sends this NCL expression in a single network message. In other words, NCL permits us to deliver the semantic power of a complex, option-laden system call through a generic multifunction expression in a single request. In the area of graphics workstations, consider an application that runs on a distant machine and sends display commands to the local system. Some of these messages may demand immediate user interaction, and others will present only a piece of the picture, requiring the user to wait until a series of RPC operations complete. In NCL, we can group related display functions into single requests. This reduces communication costs and may permit the workstation to optimize the operations through parallel evaluation or lookahead. The effectiveness of this technique was borne out by the Stanford Virtual Graphics Terminal Service system in which grouping display commands into requests proved to be a significant performance improvement [ 111. The management of an HDS requires many control messages, such as performance hints and maintenance commands. Delivering these messages as individual RPC invocations may not be a cost-effective solution in a heavily used network, yet we would like to ensure the delivery of such information in a timely manner. With NCL, we can append ancillary control information to an expression as additional functions. Typical networks confine the control information to a fixedsize header, thus limiting the number of functions. NCL expressions provide convenient opportunities for the exchange of tuning hints between clients and servers, since one never runs out of bits as one might with a conventional fixed format request-one merely appends another function to the NCL expression. 3. SPECIFICATION The Network Command Language is a special-purpose programming language for the interchange of algorithms and data between clients and servers in a distributed system. The NCL syntax is the format for specifying expressions in request and response messages. One may construct complex data and control structures in these expressions through the use of lists, vectors, and composition of functions. In the following sections, we will describe the expressions and constituent elements that are the core of NCL. 3.1 Expressions An NCL expression is a collection of one or more elements. NCL represents all elements using a uniform descriptor notation based on compact list encoding ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language
l
337
techniques [3]. NCL descriptors encode class, type, size, and structure information for the expression. The NCL expression layout provides for efficient generation and evaluation with only a 32-bit word-addressed architecture and no elaborate addressing modes. NCL aligns all descriptors and data on 32-bit word boundaries to simplify the overall addressing architecture. In addition, the entire NCL expression structure may be manipulated as a vector of 32-bit words for more efficient access. Within the descriptors, NCL aligns all fields on byte boundaries, thus avoiding potentially expensive bit-field extractions. There is theoretically no limit to the size of an NCL expression. Functions for compound structures allow the construction of large expressions, but unlike other languages, NCL must potentially return values for each of the enclosed expressions. To make this possible, each NCL expression may designate certain elements as return parameters. So although control follows a functional organization, the flow of data through an NCL expression is via explicitly designated conduits. These conduits consist of input values from the originator, return parameters back to the originator, and variables within the expression, session, or environment. An NCL request returns values via a response expression. The first element in the response expression is the status of the entire request, providing a preliminary indication of the need for further error processing. The rest of the expression may contain functions that copy values into the user space or direct the client to perform certain tasks. For example, a server can program a response expression to instruct a client to handle an unexpected situation, such as a server error or a redirected request. We can demonstrate the advantages of active responses in the case of a multiple-level storage hierarchy with a number of cooperating file servers. In this environment, the clients funnel all requests to a local file server, which may in turn have to redirect some to remote servers in the hierarchy. If we use a strict forwarding approach, then the local server must buffer and retransmit the communication between the system and the remote server. This store-andforward action places an unnecessary burden on the local server. Also, the security model may prohibit third-party intervention since it is a source of impersonation. A client can avoid these problems by communicating with the other server directly (albeit at some penalty for crossing bridges or gateways). However, it is unreasonable to expect every system to know the topology and passwords of the storage hierarchy. Instead, we can leave that responsibility with the file servers through the use of a mechanism for securely supplying this information to clients. Upon receipt of a request that should go to another server, the local server can program the requester with the information necessary to use the remote server. Then the client can send the request directly to the remote server and present the credentials programmed by the local server. 3.2 Element Classes
An NCL element belongs to one of six classes: input value, return parameter, local variable, session variable, environment variable, or reference. The class determines the NCL element descriptor format used for the representation of an ACM Transactions on Computer Systems, Vol. 5, No. 4, November 1987.
338
l
Joseph FL Falcone 8
0
class
16
31
sire location or offset
type identification genenc clemenr dcswpror
class We size identifier location otTset
unsigned e-bit value unsigned e-bit value unsigned 16-bit quantity unsigned 16-bit element signed twos-complement slgned twos-complement Fig. 3.
(bytes for vector, elements for list & function) identifier i6-bit variable location 16-b;: descriptor-relative byte-displacement
Element descriptor
formats.
object. The class names are from the perspective of the NCL evaluator; for example, an input value is an input to the evaluator, and a return parameter designates data destined for the generator of the expression. Figure 3 shows the format of NCL element descriptors for these classes. Each descriptor is a 64-bit word composed of eight g-bit bytes following a low-bytefirst organization. The first two bytes of a descriptor are unsigned values indicating the class and type of an object, respectively. This accommodates up to 256 classes of 256 types for NCL elements. The third part is an unsigned 16-bit value for the size of the element. The next two 16-bit words are either used together to form a signed 32-bit value or separately as identification or offsets to the data. Input Value. An element transporting data in an expression is an input value. Input values differ from the other classes in that actual data accompany the descriptor in the request. The descriptors of input values vary according to their type; Section 3.3 details the differences. The data of an input value are constant. Return Parameter. A return parameter establishes the nature of a result to be returned from an NCL expression. As such, it does not directly describe something that exists in the NCL expression. Instead, the evaluator of the expression uses it as a specification from which to construct an input value for the response. In this way, the request describes the minimal correct response in terms of types and sizes of input values. The second half of the descriptor contains two 16-bit unsigned values for identification. The identification area uniquely names the value in the representation of the response. Local Variable. A local variable is an element that will contain values during the execution of an NCL request or response expression on the remote system. The second half of the descriptor contains two 16-bit unsigned values for identification. The identification area uniquely names the variable in the expression. A local variable may hold an input or an output value for a function. Session Variable. A session variable is an element that contains values associated with a session between a user and a server. The second half of the descriptor contains two 16-bit unsigned values for identification. The identification area uniquely names the variable in the session. A session variable may hold an input or an output value for a function, and access may be restricted. ACM Transactions on Computer Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language
l
339
Environment Variable. An environment variable is an element that contains values associated with a server. Environment variables represent the state of a server. One alters the behavior of a server by changing the appropriate environment variables. Because they are associated with server management, a user may not view or change certain environment variables without establishing special privileges for the session. Some environment variables may be read-only for all users. The second half of the descriptor contains two 16-bit unsigned values for identification. The identification area uniquely names the variable in the environment. An environment variable may hold an input or an output value for a function, and access may be restricted. Reference. A reference is a self-relative pointer to an element within an expression. References permit the efficient representation of expressions that use the same element in several places. References conserve space in expressions at the expense of additional address indirection. 3.3 Element Types Within the classes, there are six predefined base types: list, function, link, integer, byte vector, and nil. At local option, an implementation may include an optimized special type within the existing set or add more complex types to support specialized services. In any case, clients and servers must subscribe to certain conventions for canonical data formats used as common currency in the HDS. These items include resource names, attribute designations, and wild card expressions. It is not possible to specify the format of these items as a type in most programming languages, since the description is a grammar unto itself. For example, the canonical data format for a file pathname would include the use and meaning of the various delimiters (colon, bracket, slash, dot, etc.) and strings in the pathname that NCL represents as a byte vector. Since NCL is a framework used to interface distributed system components, we chose to leave the specification of these higher level items up to the implementors of the HDS. For the input value class, there are special descriptor formats for certain types as shown in Figure 4. These descriptors accommodate the different kinds of data used in expressions. List. The fundamental data structure in NCL is the list. One uses lists to describe collections of items. There are no restrictions on the types of list elements, so lists can represent aggregate data structures. NCL expression generators construct lists in a constrained fashion, and these lists do not undergo significant transformations once built. The physical representation takes advantage of this attribute to NCL lists. Since NCL represents each element with a descriptor, a list is simply a contiguous set of descriptors. One can extend a list by making the final element a link to another set. The descriptor contains a 16-bit value for the total number of elements in the list. For linked lists, a 16-bit value in the second half of the descriptor is the number of elements in the first segment. The size information eliminates the need for a terminating nil cell and permits easier manipulation of the lists. The remaining 16 bits are the self-relative byte offset to the set of element descriptors. The physical representation of a contiguous ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
.
340
Joseph FL Falcone
0
16
8
input value
4
list
total sire offset to elements
input value segment
size
hsrdescrrptor
input
(a...
65535elemrm
funcnondescrrptor
aemoforsdf
offset1
~~,~~~~~~~~~~~~~:~~a~~~~~~~~\~ \, \\\A 1.1,. \ \. \ \\\ library name
1 function value function name
input value
31
integer
(0...65534
argunenf aescnptorr
vector byte
vector
dercnDtor
follow)
size (0 . ..65535
odtd
bytes
dt offset)
input value offset to e1ement.s
segment size lrnk descriptor
m calue
offset SiZP
(O...6553~erements
at offset)
Must Be Zero (reserved to Digital) slgned twos-complement 32-bit value slgned twos-complement 16-bit descriptor-relative byte-displacement unsigned 16-bit quantity(number of bytes or elements) Fig. 4.
Input value descriptor
formats.
n-element list is comparable to an array of n descriptors. A list may contain from 0 to 65,535 elements, and all elements not explicitly assigned must contain nil. To extend a list, one changes the last element to a link and points this link to another set of descriptors. Each link has a 16-bit field containing the size of its segment. Function. NCL functions are lists whose first element is of type function. The second half of a function descriptor contains a 16-bit library name and a 16-bit function name. Up to 65,534 arguments may follow the function element in the list. These arguments may consist of an arbitrary collection of input parameters, return values, local variables, or other functions. A function template database defines the contents of a library and the programmatic interface of each function. Each client and server fetches relevant templates upon connection to a service or at service start-up. The preamble of an NCL expression contains the version number of the template database used by its generator so that the evaluator may synchronize to it during execution. The templates specify arguments using the standard NCL classes and types, plus two additional types: value and any. The evaluator calls a function only if it is necessary to avoid a type mismatch error. In an NCL expression, an element of type function will match either function or any in the template and thus pass through without evaluation. If the template specifies another type, the evaluator will call the function and check the resulting value against it. ACM Transactions on Computer Systems, Vol. 5, No. 4, November 1987.
A Programmable Interface Language HDS Service
file system services in HDS name translation servrces in HDS error logger for HDS
Name ADMIN-FSl DEVO-FS NAMSERVl NAMSERVZ ERRLOG
Server
Network
file server for administration group file server for development group primary name server secondary name server error logger
Address DECnet node address TCP/IP address Ethernet address
10.150 192.5 58.3 AAOOOA0096 10, s
Library
Name standard NCL function library used by all services file service function library name serwce function library error log service function library
NCL FS NS ELS
Function
341
Identifier
file service flame service error log service
Server
-
Name prog, open, add, boot,
set, return,. .. read, close,. . . translate, remove,. log, alert,. . .
Fig. 5.
..
standard NCL function library used by all services file service function library name service function library error log service function library
Example components
of an NCL function
specification.
This interface includes the expected types of the arguments, the security level, the transaction semantics, and the behavioral relationship to the network communication protocol. The security level defines what class of users have access to each function. The transaction semantics indicate what action is to be taken if a function fails as part of an atomic action. Finally, the template can specify the execution of certain protocol control actions as part of the evaluation, for example, beginning a session. The full specification of a function in the HDS consists of the network host address, the service identifier, the library name, and the function name. For resources associated with a particular host, there is an additional server name specifier. Figure 5 provides examples of these specifiers. The first two components of the specification are implicit in the network message by virtue of its being sent over a particular session to a server. First, the client executive maps the system call to a particular HDS service identifier and, if the resource is server dependent, a server name. When a client process initiates a session with a server, it uses the name service to translate either the server name or service identifier to a network node address for the server host machine. Then a transport dispatch mechanism on the server host uses the service identifier to deliver messages to the appropriate server process. If a server host fails and another resumes its duties, the client can ask the name service for the network node address of this new server and reestablish the sessions to it. If a server process fails and another resumes its duties, the server host transport dispatch mechanism will deliver messages to the new server ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
342
.
Joseph R. Falcone
process, transparent to the client. Upon arrival at the destination process, an NCL evaluator calls functions in various libraries as it interprets the expression. The evaluator confines the expression to those libraries accessible to the service as determined by the database. Link. The link element allows an NCL generator to extend physically contiguous list structures into linked lists. When one extends a list, the link replaces the last element in the list, and the generator allocates a segment containing additional element descriptors for the extension. A link descriptor contains a 16-bit value for the number of elements in its segment and a 16-bit self-relative offset pointing to the segment. Integer. The integer type represents a signed twos-complement low-byte-first 32-bit value. Although we could represent the same value as a 4-byte vector, we felt that the savings in space and the establishment of a standard integer representation were sufficient reason for the distinction. Integers may also represent scalar or Boolean values, depending upon the application for interpretation.
Byte Vector. An NCL byte vector is a degenerate list, consisting of a raw collection of &bit bytes. The descriptor format is similar to that of lists, except that the size field in the header is an unsigned 16-bit value that specifies the number of bytes in the vector. Two bytes are available in the header for special information. A byte vector can contain anywhere from 0 to 65,535 bytes. However, because byte vectors contain unstructured data, one cannot arbitrarily extend a byte vector using links. One can use a byte vector as a default type to represent structures beyond the scope of NCL. Applications that must operate at a higher level than the data structures offered by NCL can use the byte vector as a raw data transport. For example, it is possible to use a byte vector to transport executable code between systems. Nil. Elements of type nil do not exist, strictly speaking. They are used to communicate the absence of an appropriate element in a list or function. 4. STANDARD
FUNCTION
LIBRARY
NCL groups all functions into libraries dedicated to particular services. All functions specified in an expression must come from libraries available to the service. The standard NCL function library is a small collection of primitives provided as a framework for constructing expressions. Each HDS service extends the standard library of functions with its own specialized libraries and canonical data formats. A complete service library consists of functions implemented on the server for requests and on the client for responses. All functions return a value that the server can process locally or return to the client. Short descriptions of the standard functions follow: (eval expression) The eval function evaluates the given expression. This function provides a
separate entry point into the expression evaluator. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language
l
343
(apply
function expression [expression 1%) The apply function evaluates a function with the given argument expressions.
(prog expression [expression]*) The prog function groups a list of expressions into a single expression. The expressions in this list execute sequentially, but prog does not guarantee atomic
execution. (atomic
expression [expression]*) The atomic function groups a list of expressions into a single expression with
the additional feature of atomic execution. If an error occurs during execution, the evaluator queries each expression in reverse order for its rollback semantics. We currently specify these rollback semantics as shadow functions for each destructive operation. Thus atomic guarantees all-or-nothing execution of a list within the bounds specified in the function library. (parallel
expression [expression]*)
The parallel function initiates evaluation on a collection of expressions simultaneously. This function presupposes the availability of multiple processors for the parallel execution. The ability to share variables between functions executing in parallel depends on the architecture of the multiprocessor. There is an implicit join at the end of the list of expressions; that is, evaluation of the expression following the parallel does not take place until the last function in the parallel expression completes. If such is not available, the list of expressions executes serially. (return
expression [expression]*)
The return function delivers values back to the generator of an expression and copies them into prespecified locations in the process address space. When results need additional processing before copying, they are passed as arguments to response functions that will perform the appropriate operations. (set expression1 expression2) The set function is the assignment operator in NCL. NCL restricts assignment
to the following classes: Input values and variables can be sources and return parameters and variables can be destinations. Any type may participate in an assignment during the evaluation of an NCL expression. (if expression1 expression2 expressions) The if function provides conditional
evaluation for one of two alternative expressions, depending on the result of a third expression. This provides a mechanism for selecting control paths within an NCL expression. In conventional RPC protocols, code running on the client always selects control paths, but in NCL the conditional statement executes on the server. (while
expression1 expression:!)
The while function provides iterative evaluation of an expression until another expression returns a false result. This is useful for expressions that apply the same operations to each element of a large data structure. The inclusion of while ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
344
l
Joseph R. Falcone
as an NCL function saves the expense of the multiple RPC invocations generated by a loop executing on the client side. However, care must be taken to prevent the generation of infinite loops. Therefore, servers must have guard quotas to abort an expression that is consuming excessive resources. (defun name (name class & type) [(name chss & type)] * expression) The defun operation provides the means for defining a new function across the network between clients and servers. This function is the most powerful tool for dynamic NCL programming. First, when a new client enters the network, it can deliver defun requests to the HDS in order to predefine command sequences that it intends to use frequently. In such a case, the defun operations are a static, one-time event for each class of client system. Clients and servers that keep track of frequently used function sequences may issue individual defun requests to add these expressions to a function library. Because of the myriad alternatives for function definition, as NCL evolves it may be desirable to attach a scope parameter to the defun operation. NCL may limit the scope of a function to a user, a node, or a collection of clients. The scope could also have limits tied to time and connection status. The power of the defun operation demands that an HDS control its use with a security mechanism. Intentional or unintentional defun calls could alter standard library definitions, a situation that could disable the HDS. Also, excessive use of the defun operation could conceivably exceed the capacity of the template database, either globally or locally. Therefore, the HDS must control the use of defun like any other resource. (debug expression [expression]*) The debug function provides an interface to the diagnostic module of the server. It allows a client to program the debug mode behavior of the server by altering the property lists associated with the NCL evaluation debugger. This behavior is server dependent and may include command logging and state dumping. (declare name cZa.ss& type size) The declare function establishes a new variable of the specified class, type, and size within the expression, session, or environment. (index vector expression) The index function selects the value in the vector indicated by the expression. (noop expression [expression]*) The noop function ignores its parameters and does nothing a placeholder for expressions that require a function call. 5. IMPLEMENTATION
at all. It serves as
CONSIDERATIONS
A heterogeneous distributed system could make heavy use of NCL expressions. Therefore an NCL implementation should make the cost of using these expressions grow incrementally with their complexity. This cost has the usual components of time and space. In many situations, we and others have observed that ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable
interface Language
*
345
the resource servers can become CPU saturated long before exhausting network bandwidth [12]. In an airline reservation system case study, a representative of the airline stated that he did not think it was still important to heavily optimize messages in order to keep costs down (the interairline protocol uses 6-bit characters for channel efficiency) [6]. Therefore, we have sacrificed some space in requiring descriptor fields to occupy full 32-bit words and &bit bytes aligned on 32-bit and &bit boundaries, respectively. This substantially reduces the cost of generation and evaluation, thereby increasing the maximum request throughput of an NCL-based server. However, this is only one of many techniques we use to efficiently implement NCL. The memory used for the construction of NCL expressions needs to be efficiently allocated, extended, and deallocated. One way to simplify the problem is to divide the expression into separate spaces for descriptors and data, in particular, bulk data. By doing this, we can use more optimal schemes for managing each space. For example, an NCL generator could avoid copying bulk data into an expression buffer by employing a scatter/gather mechanism. Using this mechanism, the generator separates the process of constructing descriptors from that of copying vector data into the messagespace. The generator can place the source address for the data in the second word of a vector descriptor and append the address of this descriptor to a list of deferred-copy elements. When the message is complete, the generator hands the descriptors along with the deferred-data addresses to the transport. The transport gathers the deferredcopy data into the message and appends additional packets as needed. However, this can have some interesting side effects. If the actual message delivery is asynchronous, one cannot allocate deferred-copy data as a local variable during generation because the data need to be accessible beyond the scope of the generator. In addition, there must be some mechanism driven by the successful delivery of a messageto deallocate memory used for expression of local variables. The packet-size restrictions of Ethernet and other networks can complicate memory management further. Because NCL uses self-relative offsets, an expression that spans many packets can lead to some complex offset calculations. In our implementation, we simplify this problem by separating the descriptors from the bulk data and restricting them to the first Ethernet packet. In order to prevent the layout of a large vector early in the expression from pushing later descriptors out of the first packet, we have a vector-size threshold that defers the placement of vector data areas until the user specifies the entire expression. Like scatter/gather, the information cannot reside on the stack nor can it change until the user specifies that the expression is complete. Generating a response expression to a request poses certain implementation challenges also. All return parameters in a request become input values to functions in the response, and someone must keep track of this mapping. The simplest approach sends the framework of the response to the server along with the request. However, if the response structure is relatively complex, this approach may not be very efficient. A better implementation would maintain the identity of the response functions locally, along with a mapping of return values to the function parameters. During the construction of the request, the generator associates each return parameter with the parameter of a function in the response. ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
346
-
Joseph R. Falcone
( open ( cvtfid filename w/urn fid I mode wlrlrn-status ) (getprop ( modtime localSmodtime J I ( putprop ( acctime enoiron.time access referPnceSmode ) ) ( debug ( log lorolAmudtime I ) )
cvtlid
filename
xturn iid
Fig. 6.
modticne
lucd modtime
3cctAme
ZR”*r*”
cinw
An example of an NCL request expression
axe65
r&rewe
mode
Ins
!ocal modtime
sent to a file service.
In most cases, the function simply copies its parameters to specific locations in the process address space. When the server evaluator returns the results, the client evaluator reassociates these return parameters with the response functions and executes them. The example in Figure 6 will illustrate the generation of NCL expressions. This example is a request to a file server, and it consists of six functions from two libraries, four lists, one reference, and one local variable. The request returns two values explicitly along with the implicit global status. The prog and debug functions are from the standard NCL library and the other functions are from a hypothetical file server library. The file service uses session state to maintain information about a series of requests. For example, the open function associates its request, as well as the enclosing session, with a particular file. Subsequent operations within the expression and session, such as the getprop and putprop functions, implicitly operate on this file. This example demonstrates the use of NCL for both function calls with arguments and structured data transfer ot results. Figure 7 presents one of the many command sequences that can generate this NCL expression. Each line presents the parameters given to add an element to the expression. For example, the first entry places a prog function in the first position of the first list taking four arguments (that are initialized to nil). From this command sequence, one can use the descriptor formats in any number of encoding styles, depending mostly on the nature of the memory management. These styles differ in their placement of indirect elements, such as lists and vectors, but do not materially affect the evaluation of the expression. A generator on one system may choose to treat the placement of certain elements differently to suit its own needs. Thus, there is a substantial amount of flexibility in the generation of an NCL expression. Our current implementations use a pay-as-you-go technique for allocating space in the NCL expression. No special information is necessary prior to generation, such as the number of descriptors to be used in the expression. The ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language element
( . . . J prog ( . . . )
open (...J cvtf-d
filename tid mode status (...J getprop (... )
modtime modtime (... 1 putprop (... I acctime time (... I debug (... I log
modtime accea.4 mode
CldSS
input value input value input value input value input value input value Input value return parameter input value return parameter Input value input value input value input value local variable’ input value input value input value input value environ variable input value Input value input value input value local variable input value reference
type list function list function list function vector vector integer integer iist function list integer integer list function list Integer integer list function list Integer integer integer integer
value
location
[list,position] Il.01 Il#ll
=
5 Function { 1.3) (3,4} 3
[3#31 l2.31 12,41 Ll.31 = [4,01 [4,1l l4.21 = is.01 15.11
PA =
4 Function
i3.01
13.4
[la41
size
(library,function)
[1,21 = [2,01 D,ll 12.21 f3.11
Function {3,14} ” lfsljrfl. ” Return #l FFA810,,j Return #2 Function 20 Local #1 Function
[6,2] = [7,0] 17.11 I7,21 [1,5] = Pa01 IS,ll
WI = D,Ol D#ll 1921
(3.8)
8 Environ #12 Function
79 8 4 4 2
(3,7}
16,ol
i6.11
[7#31 [7#41
347
l
2 2 4 2 2 2 2 2 2
{1,12}
12 Local # 1 3 D.11
2 2 4 2 4
Fig. 7. Generator command sequence for NCL request expression example.
full layout of an NCL expression begins with a preamble containing a version number and relevant parameters for the evaluator, such as the number of local variables in the request and the number of return values expected in the response. The function and data elements follow this information. Part of the layout for the sample NCL expression without the preamble is in Figure 8. In this encoding style the generator employs two pointers during construction: one for the current element descriptor and the other for the next free location in the workspace. The generator allocates lists as if they were vectors of 64-bit words, and element descriptors for a list may be intermingled with integer and vector data. A system generates a list as a series of one or more segments. At the beginning, the generator allocates a list segment using a predicted size. If the list has to grow, the generator extends it by adding links and segments. The extension process can be transparent to users of NCL. An attempt to place an element beyond the end of a list can trigger the extension to a predefined segment size. An implementation of NCL does not have to choose between efficient support for simple or complex expressions. Our experience indicates that using NCL RFE for single-function remote procedure calls provides performance comparable to traditional RPC. One would expect this, since nearly all aspects of an NCL RFE operation are similar to a traditional RPC for single-function calls, and both RFE and RPC can benefit from the same performance improvement techniques. Both schemes marshal the function and parameters into a single network request ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
348
1-4
Joseph FL Falcone
l
I...)
Input
value
2
2 l-6 I...1
t
nnpurvalue 2
1-9
I...1
mput
2
(“put
1X 2
hrt t
200 2
value
open = 4
fs = 3
2
+2-l
value
mteger
param
meger
f-
mode = FFAQIOIR 2
SUL"S
3
return
4
cvtm
c&lid
= 14
Ifi = 3
77 bytes
.. Fig. 8.
message, execute and dispatch the NCL overhead service requests. ACM Transactions
.. Memory
layout of NCL request expression
example.
the call at the server, format the results into a response message, results at the client. is nominal compared to total operation time for typical HDS We performed measurements on a VAX-II/780 VMS system
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable Interface Language
HDS Service Open/Close
I
CIien t I
IServer I Tota/
5 2 sa
I
3.556
I
8.7%
l
349
Requests I Read/Write
I
I I
7.8% 5.2% 13.00//o
I
I I
Fig. 9. NCL processing overhead as a percentage of totalprocessing (excluding disk operation time).
using a DEUNA Ethernet interface and our custom request/response transport5 This particular implementation of NCL includes only some of the performance improvements suggested in this section. NCL generator overhead was approximately 500 microseconds for multifunction or multipacket expressions and 250 microseconds for single function expressions. NCL evaluator overhead was approximately 250 microseconds for requests or responses. Figure 9 shows the NCL processing overhead for file open/close and file read or write request and response expressions as a percentage of total processing, excluding the time used by the disk drive operations. These results agree with those presented by the implementors of Matchmaker in [7, p. 2291 and provide additional evidence in support of their conclusion: The currently generated Matchmaker code is known to be inefficient. Yet, it is important to note, as shown by the above statistics, that actual Matchmaker overheads in message passing have no perceivable effect upon system performance. If they were eliminated entirely, it would not be noticed. Although all communication in SPICE is via messages and Matchmaker, we know of no normal system activity which is dominated by message passing time. Even at the maximum message passing rate, roughly % of the total time would still be used for other things. This is easily explained, it almost always takes longer to process the information passed in messages than the time it took to transmit it, given Accent’s efficient message implementation.
For multioperation system calls NCL RFE holds a significant advantage over RPC. In NCL RFE, the multiple operations translate to multiple functions within a single expression that is sent in a single network message to an NCL evaluator that makes multiple calls on the recipient. The NCL evaluator is little more than an RPC dispatcher that can dispatch iteratively for lists of functions and recursively for nested functions as it steps through an NCL expression. For RPC, the multiple operations become multiple remote procedure calls each sent in its own message. Compared to single function calls, only the generation/ evaluation overhead increases for a multifunction NCL RFE, whereas all overheads increase for multiple RPC invocations, including the expensive network messageoverhead. Given the substantial performance difference between internal buses and computer networks, one is better off performing multiple calls on the 5 DEUNA
is a trademark
of Digital
Equipment
Corporation.
ACM Transactions en Computer Systems, Vol. 5, No. 4, November 1987.
350
l
Joseph R. Falcone
recipient’s internal bus via an NCL evaluator instead of across the computer network via an RPC. And, as noted earlier, a system may piggyback management functions onto any NCL expression, saving additional network messages. ACKNOWLEDGMENTS
Joel Emer and I arrived at the necessity for the Network Command Language independently and simultaneously. In 1984, I built the first general NCL evaluator (still in use); Rick Fadden and Rich Whalen developed routines for generating and manipulating NCL expressions. I would like to thank Linda Wright, Israel Gat, and Jeff Spirn for their support during their tenure as managers over the project. Karri Lynne Kaiser patiently reviewed drafts of this document as the NCL concept evolved. The referees suggested many inprovements that resulted in a clearer presentation of the concepts. Finally, I reserve special thanks for our technical operations group, without whom our superb prototype environment would not have been possible: Walt Ronsicki, Pat Hunt, Jeff Dickens, and Bob Weithman. REFERENCES 1. ADOBE SYSTEMS, INC. PostScript Language Reference Manual. Addison-Wesley, Reading, Mass., 1985. 2. BIRRELL, A. D., AND NELSON, B. J. Implementing remote procedure calls. ACM Trans. Comput. Syst. 2, 1 (Feb. 1984), 39-59. 3. BOBROW, D. G., AND CLARK, D. W. Compact encodings of list structure. ACM Trans. Comput. Syst. 1, 2 (Oct. 1979), 266-286. 4. BROWNBRIDGE, D. R., MARSHALL, L. F., AND RANDELL, B. The Newcastle connection or UNIXes of the world unite! Softw. Pratt. Exper. 12 (1982), 1147-1162. 5. DIGITAL EQUIPMENT CORPORATION. Ethernet, a local area network: Data link layer and physical layer specifications version 1.0. DEC, Littleton, Mass., Sept. 1980. 6. GIFFORD, D., AND SPECTOR, A. The TWA reservation system. Commun. ACM 27,7 (July 1984), 650-665. 7. JONES, M. B., RASHID, R. F., AND THOMPSON, M. R.
8. 9. 10. 11. 12. 13. 14. 15. 16.
Matchmaker: An interface specification language for distributed processing. In Proceedings of the 12th Annual ACM Symposium on Principles of Programming Languages (New Orleans, La., Jan. 14-16). ACM, New York, 1985, pp. 225-235. KENAH, L. J., AND BATE, S. F. VAX/VMS Internals and Data Structures. Digital Press, Bedford, Mass., 1984. KNIGHT, J. R. A case study: Airlines reservations systems. Proc. IEEE 60, 11 (Nov. 1972), 1423-1431. A closely-coupled distributed KRONENBERG, N., LEVY, H., AND STRECKER, W. VAXclusters: system. ACM Trans. Comput. Syst. 4,2 (May 1986), 130-146. LANTZ, K. A., AND NOWICKI, W. I. Structured graphics for distributed systems. ACM Trans. Graph. 3, 1 (Jan. 1984), 23-51. LAZOWSKA, E. D., ZAHORJAN, J., CHERITON, D. R., AND ZWAENEPOEL, W. File access performance of diskless workstations. ACM Trans. Comput. Syst. 4, 3 (Aug. 1986), 238-268. LYON, B. Sun external data representation specification. Tech. Rep., Sun Microsystems, Inc., Mountain View, Calif., 1984. MCCARTHY, J., ET AL. LISP 1.5 Programmer’s Manual. M.I.T. Press, Cambridge, Mass., 1962. METCALFE, R. M., AND BOGGS, D. R. Ethernet: Distributed packet switching for local computer networks. Commun. ACM 19,7 (July 1976), 395-404. NATIONAL BUREAU OF STANDARDS. Data Encryption Standard. FIPS Publ. 46, NBS, U.S. Department of Commerce, Washington D.C., Jan. 1977.
ACM Transactions
on Computer
Systems, Vol. 5, No. 4, November
1987.
A Programmable
Interface Language
l
351
17. NEEDHAM, R. M., AND HERBERT,A. J. The Cambridge Distributed Computing System. AddisonWesley, London, 1982. 18. NELSON, B. J. Remote procedure call. Ph.D. dissertation, Carnegie-Mellon University, Pittsburgh, Pa., 1981. Also available as CMU Tech. Rep. CMU-CS-81-119. 19. RITCHIE, D. M., AND THOMPSON, K. The UNIX time-sharing system. Commun. ACM 17, 7 (July 1974), 365-375. 20. SHELTZER, A. B., AND POPEK, G. J. Internet Locus: Extending transparency to an Internet environment. IEEE Trans. Softw. Eng. SE-12,ll (Nov. 1986), 1067-1075. 21. THOMAS, R. H. A resource sharing executive for the ARPANET. In Proceedings of the 1973 AFZPS National Computer Conference (June 4-8, New York, N.Y.). AFIPS Press, Reston, Va., 1973, pp. 155-163. 22. WALKER, B., POPEK, G., ENGLISH, R., KLINE, C., AND THIEL, G. The LOCUS distributed operating system. In Proceedings of the 9th ACM Symposium on Operating Systems Principles (Bretton Woods, N.H., Oct. 11-13). ACM, New York, 1983, pp. 49-70. 23. XEROX CORPORATION. Courier: The remote procedure call protocol. Xerox System Integration Standard XSIS-038112, Xerox Corporation, Stamford, Conn., Dec. 1981. 24. XEROX CORPORATION. Interpress: Electronic printing standard. Xerox System Integration Standard XSIS-048404, Xerox Corporation, Stamford, Conn., Apr. 1984. 25. ZIMMERMANN, H. OS1 reference model-The IS0 model of architecture for open systems interconnection. IEEE Trans. Commun. COM-28,4 (Apr. 1980), 425-432. Received April 1986; revised December 1986; accepted June 1987
ACM Transactionson ComputerSystems,Vol. 5, No. 4, November1987.
