Typing Erlang - CiteSeerX

Typing Erlang John Hughes, David Sands, Karol Ostrovsk´y December 12, 2002

Abstract Type systems for concurrency express vital properties of concurrent and distributed programs, such as deadlock-freeness. These systems have grown more sophisticated with time, but unfortunately most of them apply to toy programming languages. At the other end of the spectrum, there is Erlang – an open souce functional and concurrent language with a number of successful commercial applications, but lacking a good type system. Attempts have been made to define a type system for Erlang, but these have ignored its concurrency, distribution and fault-tolerance features. The goal of this project is to develop a type system for Erlang that can be used to successfully type check existing large Erlang programs, and used as a substrate for efficient type-based program analyses. Our approach is also inspired by recent developments of the Haskell type system, which now supports a number of advanced features (e.g. polymorphic recursion and existential types). Each of these features poses problems for type inference, but succumbs easily to type checking. Haskell’s design has shown that it is possible to combine explicit type annotations, for the checkable parts, while still using type inference for most of the program. By using the same approach for Erlang we hope to combine efficiency and power.

1

Introduction

The goal of this project is to develop a type system for Erlang, which can be used successfully to type check existing large Erlang programs, and as a substrate for efficient type-based program analyses.

1.1

Erlang: Background

Erlang is a concurrent functional language developed by Armstrong et al. at Ericsson, primarily for telecommunications applications. It has been used on a large scale at Ericsson, and is now found in a number of products, including a million-line system controlling the AXD 301 ATM switch, currently Ericsson’s most profitable. The Erlang “Open Telecoms Platform” (OTP), on which the AXD 301 system is based, is now recognised to be part of Ericsson’s core business. The language has also spread to other companies developing robust distributed applications, such as Sendmail, Bluetail (now part of Nortel Networks), and T-Mobile (the second largest mobile phone operator in the world). And in academia, the ACM’s conglomerate of conferences Principles, Logics, and Implementations of high-level programming languages has since 2001 included an Erlang Workshop. Erlang’s success is due to the combination of a number of useful features. From functional languages, Erlang inherits immutable data, powerful list operations, and first-class functions. It adds efficient, yet simple to use, support for large-scale concurrency and distribution, coupled with powerful mechanisms for fault tolerance. Using higher-order functions to capture useful abstractions, Erlang/OTP packages ideas such as generic servers, or supervisory trees for automatically restarting failing subsystems, in a form which programmers can use without needing to use concurrency and fault tolerance mechanisms explicitly.

1.2

Typing in Erlang

In contrast to most other modern functional languages, Erlang lacks a static type system. The benefits of static typing are well-known: rapid discovery of errors (“once it typechecks, it works”), and a guarantee that “well typed programs don’t go wrong”. Although Erlang programmers get by without one, and the fault-tolerance mechanisms help cover up for any residual type errors in production code, a static type system would bring clear benefits in practice. Indeed, users’ desire for a type system is a recurring topic at the Erlang Users Conference. Thus meeting the goal of this project would benefit Erlang programmers in practice. It would be easy to criticise the Erlang designers for simply missing that modern functional languages are typed — but this would be disingenuous. A number of Erlang features make a type system non-trivial. Erlang is concurrent, and typing concurrency is still an active research issue. Erlang supports persistent data via the Mnesia distributed database, which can hold objects of any type — typing persistence is also non-trivial. Erlang’s reflection mechanisms, used heavily in OTP, and hot loading of new code in running systems, also pose tricky typing problems. Each of these features is essential to Erlang’s success in its target application domain. It is simply not true to say that the Erlang designers could have taken an off-the-shelf system and used it.

1.3

Scientific Challenges

Erlang/OTP offers a real opportunity to do research which is both scientifically interesting and relevant in practice. The Erlang language itself is fairly simple and amenable to study, sharing much with both functional languages and process calculi. At the same time it incorporates “real world” mechanisms for fault tolerance and hot code replacement which are usually omitted from the toy languages academics like to study. Erlang’s design is not only proven in practice: the OTP library offers large volumes of open source code, encapsulating some of the most important design patterns, which is available for us to study. As a result there is no need to invent problems by abstracting away from details in some toy calculus: we can directly study production code for the most important kinds of application. This is a golden opportunity to study typing concurrency in practice. Retrofitting a type system to an existing language is never easy, particularly if one wants to type check most existing code without change. Marlow and Wadler have already attempted to do so for Erlang. Their goal was to type most programs without even requiring programmers to add datatype definitions (which most typed languages require), and to accept common Erlang practices such as returning a special atom, rather than a value of the expected result type, as a way to represent exceptions. Despite their emphasis on keeping the system simple and “practical”, in practice their work was a failure. Forced to use subtyping, their type-checker became unreasonably slow on inputs of more than a few thousand lines, and produced error messages that could be pages long, which real programmers could not understand. They also addressed only sequential Erlang, thus omitting perhaps the most interesting parts of the language. Our conclusion is that Marlow and Wadler were overambitious: they produced an interesting type system, rather than a well-engineered one. However, an unexpected result of their work is that the Erlang team, discovering that some common idioms were hard to type, changed their coding style to avoid them! Thus the problems that Marlow and Wadler tried to solve have now largely disappeared, which should permit a more conventional (and efficient!) type system to be used for at least the sequential parts of Erlang.

1.4

Goals

Our approach is more modest. We plan to base our work on a conventional polymorphic type system. We recognise the importance of type inference for checking legacy code — noone is going to add explicit type information to millions of lines of Erlang. On the other hand, we consider it is acceptable to add a small amount of explicit type information (notably datatype definitions) in the interests of making inference cheap, and error messages comprehensible. By

striking an appropriate balance here, we should be able to make a well-engineered type system for the sequential parts of Erlang. The real challenge is to address the non-standard parts of Erlang, particularly concurrency. This will require a more powerful type system than the standard Hindley-Milner one. Our approach is inspired by recent developments of the Haskell type system, which now supports polymorphic recursion, higher-rank polymorphism, and existential types, among others. Each of these extensions poses problems for type inference, but succumbs easily to type checking. Haskell’s design has shown that it is possible to combine explicit type annotations, for the checkable parts, while still using type inference for most of the program. By using the same approach for Erlang we hope to combine efficiency and power.

2

Related Work

Here we review some of the related work in static typing for Erlang and for concurrency in general – although we make no attempt to be comprehensive.

2.1

Erlang Type Systems

The work of Marlow and Wadler [11] was the first attempt to desing a practical type system for Erlang. Their goal was to type most programs without requiring programmers to make changes to the code. Their type-checker was unreasonably slow on inputs of more than a few thousand lines, and produced error messages that could be pages long, which real programmers could not understand. Also, they addressed only sequential Erlang, completely omitting the issues of concurrency, distribution and fault tolerance. As a result of Marlow’s and Wadler’s attempt, the Erlang team redesigned many of the libraries to allow a more conventional type system. This allowed Arts and Armstrong [2] to design a new simpler type system based on Hindley-Milner. This new system provides only type checking, and thus requires explicit type annotations. As with the earlier system, it can only type check sequential Erlang. The practical use of this type system is limited, but its annotations serve a useful standardised documentation of function types in the OTP libraries.

2.2

Concurrency Types

Considering the issues raised by concurrency, much of the recent research on typing concurrent languages centers around π-calculus [15, 16, 14, 13] and its (mainly) asynchronous derivatives (see for example [6, 18, 19]). The initial work on on both type checking and type inference (see for example [17, 23, 10]) concentrated on assigning input/output types to channels. Since Erlang processes have single mailbox that handles all communication these basic type systems are not expressive enought. An Erlang type system which deals with concurrency has to capture the structure of the communication that is happening in the mailbox. Closer to our needs are the recent result that attempt to assign more complex types or behaviours to processes abstracting multichannel communication structure [3, 8]. These type systems use simpler calculi, for example subsets of CCS [12], with certain decidable properties to aproximate a process communication behaviour. Similar work by Gay and Hole [4] and Honda at al. [7] uses so-called session types that express changing communication structure on one channel which is directly related to the single mailbox communication between Erlang precesses. The work of Amtoft, Nielson and Nielson [1], in the Concurrent ML context, shows that similar effects or behaviours can be incorporated into a Hindley-Milner type system, although here the synchronisation mechanism is rather more complex than just asynchronous communication. The planned type system should serve as a base for futher possible extensions. For example Erlang provides facilities for dynamic code update. These can potentially lead to crashes, for example if a user would load a new code that does not work with values similar to the values used

by the old code. The work of Sewell [21] shows that there might be a possibility to extend the type system to handle distributed version control. There are also a number of possible security related extensions. Most of the existing Erlang applications run on closed networks, something which is rather useful from a static typing perspective. But the possibility to run them in open (anonymous) environments exists. Our hope is that the type system should also facilitate security analyses similar to those for example in [20, 5].

3

Preliminary Results

This section gives some technical insight into some of the methods that could be called upon to tackle the challenging problem of statically typing Erlang. This work is preliminary, and is the result of collaboration with Thomas Arts, Ericsson AB.

3.1

The Generic Server

As an example of the typing problems that arise in real Erlang code, and the kind of techniques that may solve them, we shall discuss the generic server, which is a heavily used abstraction in the OTP library. The server is an Erlang process which accepts messages containing requests, processes them, and sends replies. It is generic because it is parameterised on the message types and the processing. Typing the generic server and its invocations raises a number of difficulties, which we will consider in some detail. Our purpose is not to show how to type generic servers in particular, but rather to show that typing real Erlang programs is far from trivial, and to sketch how more advanced typing methods can solve such problems as arise. loop(M,S) -> receive {call, Msg, Pid, Ref} -> case M:handle_call(Msg,S) of {reply, R, NewS} -> Pid!{reply, R, Ref}, loop(M,NewS) end; Msg -> ... end. Figure 1: The generic server main loop. The (somewhat simplified) main loop of the generic server is shown in Figure 1. It begins by receiving a message from the server’s mailbox (each Erlang process has a single mailbox containing incoming messages; there is no concept of a channel as in most process calculi). If the message is invoking a service, i.e. matches the pattern {call,Msg,Pid,Ref}, then the call is handled by the handle_call function in module M (which is passed as a parameter). The result is a reply, which is forwarded (by Pid!{reply, R, Ref}) to the process ID Pid sent by the caller — which is intended to be the caller’s own. The Ref component of the call message is a unique reference user by the caller to recognise that the reply, when it is received, is associated with that particular call message. A type system should, of course, ensure that messages sent to the server are of the type expected. We can do so by assigning each process ID a type Pid τ indicating the type τ of messages expected, and giving the message send operator the polymorphic type (!) :: ∀α. (Pid α, α) → () thus requiring that the message type matches the Pid type.

3.2

Type and Effect Systems

But how do we ensure that the receive expressions in a process body expect messages of the correct type? The Pid is not an explicit parameter of receive; instead messages are taken from the current process’ mailbox. We can capture this connection using a type and effect system [22]: let us write typing judgements in the form Γ ⊢ e : τ receiving µ where the “effect” µ is the type of messages received. Thus an expression has both a result type, and a message type associated with it. Now the type system can ensure that every receive in a process body expects the same type of message. Moreover, the process creation function spawn can return a Pid whose type matches the effect of the spawned process body.

3.3

Typing First Class Modules

Notice that the generic server loop is passed a module M as a parameter. Modules in Erlang are truly first-class citizens: they can be passed as parameters, stored in data structures, even loaded and replaced at run-time. Moreover, these capabilities are used heavily in the OTP library, so we must be able to type them. Typing first-class modules is a difficult problem. ML’s parametric module system seems to be too static for this; instead we plan to base our work on Mark Jones’ proposals for Haskell [9]. Thus we assign loop a type such as loop :: ∀µ, ρ, σ. ( sig

init :: () → σ handle_call :: (µ, σ) → reply ρ σ

end, S) → () receiving call µ ρ indicating both the signature of the module parameter, and that the messages sent to the server must have a matching type.

3.4

Existential Types

The type of messages the server expects, call µ ρ (where we use call both as the name of a type, and as a tag), represents requests containing a message of type µ, and expecting a reply of type ρ (which matches the type of handle_call in the parameter module). Referring to Figure 1 again, such an object must contain a unique reference (Ref), say of type ι, and the Pid of a process expecting messages of type reply ρ ι. We might declare the type of such an object by data call µ ρ = {call, µ, Pid (reply ρ ι), ι} But this can’t be quite right: the type ι isn’t bound! Notice, however, that the type we use for unique references is really irrelevant as far as the code of the generic server is concerned; we might even use different types in different iterations of the loop. All that matters is that the unique reference and the Pid in every call message agree on the type ι. Thus we can use an existential type as the type of call messages: data call µ ρ = ∃ι.{call, µ, Pid (reply ρ ι), ι} When we typecheck the server loop, the type ι of Ref is now independent of all other types, which means that the only way to use Ref is in a reply sent to the corresponding Pid — otherwise the types will not match. Thus, by using an existential type, we guarantee that replies are sent to the correct process. Client processes do not communicate with generic servers by explicit message passing — that would be too error-prone. Instead, they use a function call (another pun!) defined in the generic

call(Pid,Msg) -> Ref = makeref(), Pid!{call, Msg,self(),Ref}, receive {reply, R, Ref} -> R end.

Figure 2: Calling a generic server. server module (see Figure 2). It takes the server ID and message as parameters, and returns the reply as its result. Internally, it creates a unique reference using makeref() (which returns a different result on every call), and uses pattern matching in the receive to ensure that the reply taken from the caller’s mailbox corresponds to the call message sent. Since makeref returns a unique value on every call, we may wish to reflect that in the type system. We can do so by giving it the type makeref :: () → ∃ι.ref ι which, in principle, allows makeref to return a value of a different type at each call. Following Haskell’s approach, when we bind the result to Ref we “open the existential package”, thus giving Ref a type different from all others. With this approach, the type-checker cannot identify the types of two different references, and thus can check that we use the right reference at each occurrence. Now, the receive expression in call has the type receive . . . :: ρ receiving reply ρ (ref ι) where ρ is the type of the result, and ref ι the type of Ref. The entire process in which call appears receives messages of the same type, and so in particular the call of self() (which returns the process ID of the executing process) appears in a process receiving messages of type reply ρ (ref ι). By giving self the type self :: ∀µ.() → Pid µ receiving µ we can ensure that it returns a Pid expecting the matching type of message.

3.5

Typing Call

What, then, is the type of the call function itself? It is passed the ID of a server, and a corresponding message, returns the reply, and receives a message containing the reply and a unique identifier. We might therefore expect it to have the type call :: ∀µ, ρ, ι.(Pid (call µ ρ), µ) → ρ receiving reply ρ ι But there are two problems with this type. Firstly, each process may in general call several generic servers, which means that is must expect to receive replies of several different types! Indeed, a generic server may itself communicate with other servers, and thus need to accept both messages of its own request type, and replies from other servers. This violates our fundamental assumption that all messages sent to the same process have the same type; in the type system it means that different parts of a process will have inconsistent effects. The second problem is more technical: in the type of call above, the type of unique references ι is universally quantified — but in the body of call, it was existentially quantified. We should therefore give call some kind of existential type, and indeed, the most sensible place to quantify over ι seems to be in the effect. So let us instead give call the type call :: ∀µ, ρ.(Pid (call µ ρ), µ) → ρ receiving ∃ι.reply ρ ι

Now, the type ι has become local to the effect — it does not appear in the arguments or results of the function. It follows that no external code can “manufacture” a message of the correct type to send to call. (The generic server loop can do this only because call itself provides it with the “capability”). Thus we can consider that, for all practical purposes, call receives nothing. We introduce sub-effecting, and allow this type to be generalised to call :: ∀µ, ρ, ǫ.(Pid (call µ ρ), µ) → ρ receiving ǫ which now permits call to appear in any process, regardless of the type of message it normally expects. What we have done is shown that communication with a generic server can be considered independently of the other communication a process engages in. Of course, it is the purpose of the unique references to make this possible. We introduced existential types for unique references just in order to reflect that in the type system, and make typing generic server calls possible.

3.6

Reflections

The ideas in this section are far from a complete type system for generic servers — there remain several details to work out. But what we hope we have shown, is that a combination of ideas such as types-and-effects, sub-effecting, and existentials can be used to approach difficult typing problems that arise from real Erlang code, with a good expectation of success.

References [1] Torben Amtoft, Flemming Nielson, and Hanne Riis Nielson. Type and Effect Systems: Behaviours for Concurrency. Imperial College Press, 1999. [2] Thomas Arts and Joe Armstrong. A practical type system for erlang. Erlang User Conference, 1998. [3] Sagar Chaki, Sriram K. Rajamani, and Jakob Rehof. Types as models: Model checking message-passing programs. In The 29th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 2002. [4] Simon J. Gay and Malcolm Hole. Types and subtypes for client-server interactions. In European Symposium on Programming, pages 74–90, 1999. [5] Matthew Hennessy and James Riely. Type-safe execution of mobile agents in anonymous networks (extended abstract). In J.Vitek and C.D. Jensen, editors, Secure Internet Programming, volume 1603. Springer-Verlag, July 1998. [6] Kohei Honda and Mario Tokoro. An object calculus for asynchronous communication. In ECOOP’91 LNCS 512. Springer-Verlag, 1991. [7] Kohei Honda, Vasco Thudichum Vasconcelos, and Makoto Kubo. Language primitives and type discipline for structured communication-based programming. In European Symposium on Programming, pages 122–138, 1998. [8] Atsushi Igarashi and Naoki Kobayashi. A generic type system for the pi-calculus. In The 28th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 2001. [9] Mark P. Jones. Using parameterized signatures to express modular structure. In The 23rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 1996.

[10] Kabayashi, Benjamin C. Pierce, and David N. Turner. Linearity and the pi-calculus. In The 23rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 1996. [11] Simon Marlow and Philip Wadler. A practical subtyping system for Erlang. In International Conference on Functional Programming, pages 136–149. ACM, June 1997. [12] Robin Milner. Communication and Concurrency. International Series in Computer Science. Prentice Hall, 1989. [13] Robin Milner. The polyadic π-calculus: a tutorial. Technical Report ECS-LFCS-91-180, Department of Computer Science, University of Edinburgh, October 1991. [14] Robin Milner. Communicating and Mobile Systems: the π-Calculus. Cambridge University Press, 1999. [15] Robin Milner, Joachim Parrow, and David Walker. A calculus of mobile processes, Part I. LFCS Report Series ECS-LFCS-89-85, University of Edinburgh, Jun 1989. [16] Robin Milner, Joachim Parrow, and David Walker. A calculus of mobile processes, Part II. LFCS Report Series ECS-LFCS-89-86, University of Edinburgh, Jun 1989. [17] Benjamin C. Pierce and Davide Sangiorgi. Types and subtypes for mobile precosses. In Proceedings of the IEEE Conference on Logic in Computer Science. IEEE Computer Society Press, 1993. [18] Benjamin C. Pierce and David N. Turner. Pict: A programming language based on the pi-calculus. CSCI Technical Report 476, Indiana University, 1997. [19] James Riely and Matthew Hennessy. A typed language for distributed mobile processes. In The 25th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. [20] James Riely and Matthew Hennessy. Trust and partial typing in open systems of mobile agents (extended abstract). In The 26th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1999. [21] Peter Sewell. Modules, abstract types, and distributed versioning. In The 28th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, January 2001. [22] Jean-Pierre Talpin and Pierre Jouvelot. The type and effect discipline. Information and Computation, 111(2):245–296, 1994. Extended abstract in Proceedings of the IEEE Conference on Logic in Computer Science (LICS’92). IEEE Press, June 1992. [23] David N. Turner. The Polymorphic Pi-Calculus: Theory and Implementation. PhD thesis CST–126–96, Dept. of Computer Science, University of Edinburgh, 1996.