Micro-Reboot Support for Multi-Server Operating ...

Micro-Reboot Support for Multi-Server Operating Systems Hiroo Ishikawa Tatsuo Nakajima Department of Computer Science, Waseda University {ishikawa, tatsuo}@dcl.info.waseda.ac.jp Abstract

ware for future sensor nodes should be made of multiple cooperative processes for security reason. Some research projects show a secure construction of an Internet server where it is divided into multiple processes depending on access privileges to data [3][4]. As for sensor nodes connected to the Internet, not only the applications, but also the operating system should be constructed in that way. Failure notification is another important feature of a reliable operating system. In the situation where every application is constructed of multiple cooperative processes, a failure on a process may break the consistency of the application. The failure notification gives an opportunity for the other processes to deal with the failure. The current operating systems tend not to notify a failure of a process to the other processes. Failure notification is necessary for a reliable operating system such as a multi-server operating system. We propose a multi-server operating system that supports micro-reboot (uRB) as the first class abstraction. uRB is a technique originally proposed for Internet servers to improve service availability by restarting a part of a system which suffers from a failure [5]. Its core idea is to improve availability by reducing the time for recovery by rebooting a part of a system rather than increasing the available time. Unlike the original idea of uRB, there are several challenges to apply uRB at the operating system level. While a web application server is clearly separated to application logic and data, a process is not separated to logic and computational state because it is a more general abstraction than a web application server. Consequently, the operating system kernel is required to preserve process state over its restart for the system consistency. First, the kernel should provide the mechanism to notify the restart of a process to the other running processes. Second, the kernel should provide the notion of persistent memory or continuations. This paper is structured as follows. The next section discusses the key principles of ArcOS. Then, the implementation of ArcOS is described in Section 3. Section 4

Self-healing is an important property for computing nodes that are required to operate for a long time without dedicated administrators such as sensor nodes. The current operating systems for such computing nodes provide insufficient self-healing support; they lack failure containment and/or failure notification for their subsystems and user tasks. The former prevents a whole system from crashing, and the latter is necessary for recovering the failure. We adopt microkernel architecture for the former requirement and propose failure notification support for the latter.

1

Introduction

Sensor nodes are required to operate for a long time without any dedicated administrators. Some of them are connected to the Internet for delivering local weather monitoring etc. [1]. Some other aggregate and relay the data from other wireless sensor nodes [2]. Even some of recent or future consumer electronics are equipped with sensors and wireless connectivity. Those sensor nodes are assumed to run 24/7 and required to handle failures by themselves. The current operating systems, however, make few efforts for availability; a whole system is simply terminated if it suffers from a failure. All the services provided by the node are also terminated. Failure containment and notification are the facilities to improve the availability of the sensor nodes. Failure containment is realized by protection domains, which logically separate memory regions by means of hardware (i.e MMU) or software tools (e.g. compilers) so that the operating system kernel can detect invalid accesses over domain boundary due to errors. However, many software systems on a sensor node are constructed as monolithic programs. It should be divided to multiple protection domains in terms of its reliability. Moreover, we believe that soft1

shows some related work. Then we conclude the paper in Section 5.

2

In contrast, a node is removed from a tree when the process is terminated or explicitly disconnect from its server process. A dependency tree is accessible from any process in it. It maintains the state of the processes in it so that a process that has received a notification can make a decision of recovery depending on the decisions of the dependent processes.

OS Support for uRB

The core of our uRB support is to find the smallest collection of processes, called a restart group, to recover a failure. We use restart notifications to form a restart group. Each process that has received the notification invokes its event handling procedure to join the current restart group. The processes in a restart group are restarted at the same time by the kernel. This restart may trigger another restart notification (Figure 2.2). The kernel maintains dependency trees and process state to realize the uRB procedure described above. A dependency tree represents the relationships between processes. A restart notification is delivered along with the dependency trees. The process state machine expresses the state transitions corresponding to uRB. The kernel controls the delivery of a notification by means of the state machine.

1st generation

2nd generation

2.2

A process can keep its consistency with the process to be restarted by receiving a notification of the restart. For instance, the process receiving the notification may immediately close the connection to the crashed process, or it may keep the connection after the process restarted and send data again for the process so that it recovers its pre-crash state. In case one of the processes decides to restart in response to the notification, another notification of the restart of the process is generated and delivered to the sub-tree. Figure 2 illustrates the delivery of a notification. A delivery of notification wakes up the main thread of a process. The notification handler is invoked by the main thread upon a restart notification. The notification handler is allowed only to make a decision for joining or leaving the current restart group so that the system can form a restart group immediately after a failure detection. Depending on the decision, another notification is delivered to the next process.

3rd generation

Figure 1. A dependency tree and the propagation of restart notifications. The arrows represent the notifications of restart. The gray circles represent the processes restarting, which join a restart group. The white circles represent the processes not restarting. The dotted circle represents the restart group in the first generation. A notification is propagated if a process restarts.

2.1

Propagation of Restart Notification

1. notification

3. notification

2. handling

4. handling

Figure 2. Notification delivery and the order of its procedure.

Dependency Trees

3 A dependency tree represents the relationships between processes in order to propagate the failure and restart of a process. We assume that any two processes that interact with each other have a client-server relationship. A server process is always a parent node of the client node in a dependency tree. The kernel expands a dependency tree when a client process establishes a connection to a server process.

ArcOS

ArcOS is a multi-server operating system based on the L4 microkernel [6]. Operating system services such as device drivers, file systems, etc. run in user mode as other user tasks do. Thus, each OS service is isolated from each other and from the kernel. The kernel analyzes the dependencies between process to find the subset of the whole system to be micro-rebooted. 2

5

The current implementation depends on the communication abstractions provided by the L4 microkernel. Since IPC and shared memory are the ways processes interact with each other, the privileged task, called Arc root task, watches them to maintain dependency trees. Arc root task (ART) is in charge of memory mapping control in addition to task lifecycle control such as thread creation/deletion and interrupt delivery to user tasks. If two tasks attempt to share a memory region, ART maps certain number of pages to both tasks’ address spaces. At this moment, client-server role of the tasks is determined: a task that has initiated page sharing becomes a client, and a task that has accepted page sharing becomes a server. Page mapping relationships are stored in the memory mapping database maintained by ART. Upon a failure of a task, ART looks through the memory mapping database and finds server tasks of the task. Then, ART propagates a failure notification to form a restart group as described in Section 2.2.

4

Concluding Remarks

Micro-rebooting enables a system to heal itself. Failure notification supported by an operating system identifies the group of processes that would be affected by the failure and is necessary for micro-rebooting of a system. This paper has shown the dependency tree and the notification delivery as technologies to realize the failure notification inside a system. We are currently implementing these features on a multi-server operating system.

References [1] Hiroshi Esaki and Hideki Sunahara. Live E! Project; Sensing the Earth with Internet Weather Stations. In Proceedings of the 2007 International Symposium on Applications and the Internet (SAINT’07), January 2007. [2] Omprakash Gnawali, Ben Greenstein, Ki-Young Jang, August Joki, Jeongyeup Paek, Marco Vieira, Deborah Estrin, Ramesh Govindan, and Eddie Kohler. The Tenet Architecture for Tiered Sensor Networks. In Proceedings of the 4th ACM Conference on Embedded Networked Sensor Systems (SenSys’06), October 2006.

Related Work

ChorusOS recovers an actor (a process under ChorusOS) from a failure by restarting it [7]. To preserve the state of actors, it provides persistent memory to them. To recover the latest state of an actor, it makes checkpoints of it. Chorus also defines a restart group to manage a set of actors that are connected by dependencies. As a result, restart groups form a restart hierarchy. A restart action is propagated in a way that a parent of the restart group tree takes in charge of its children. If the parent fails to obtain normal response from its restarted children, the grand parent restarts the parent. Chorus provides some additional primitives for propagation, such as the types of restart actions. Nooks gives a separate address space to each Linux driver so that it is isolated from the kernel and restart independently [8]. The shadow driver is provided to take over a device driver during its reboot in order to prevent applications that have used the device driver from crashing because of its short absence. CuriOS is a research operating system that support process restarting [9]. Its operating system services and user programs are written in C++ so that they can receive system errors such processor exceptions or memory violation through C++ exception mechanism. A server is allowed to store a part of its state corresponding to a client to the client’s address space so that the server can recover the state after a restart.

[3] Niels Provos, Markus Friedl and Peter Honeyman. Preventing Privilege Escalation. In Proceedings of the 12th USENIX Security Symposium. August 2003. [4] Maxwell Krohn. Building Secure High-Performance Web Services with OKWS. In Proceedings of the 2004 USENIX Annual Technical Conference (USENIX’04), June 2004. [5] George Candea, Shinichi Kawamoto, Yuichi Fujiki, Greg Friedman, and Armando Fox. Microreboot – A Technique for Cheap Recovery. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI’04), December 2004. [6] L4 eXperimental Kernel Reference Manual, Version X.2, System Architecture Group, Department of Computer Science, Universit¨at Karlsruhe, November 2006. [7] Vadim Abrossimov, Frederic Hermann, Jean Christoph Hugly, Frederic Ruget, Eric Pouyoul, and Michel Tombroff. Fast Error Recovery in CHORUS/OS: The Hot-Restart Technology. Technical Report, Chorus Systems, Inc. 1996. 3

[8] Michael Swift, Muthkaruppan Annamalai, Rian Bershad, and Henry Levy. Recovering Device Drivers. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI’04), December 2004. [9] Fransis M. David and Roy H. Campbell. Building a Self-Healing Operating System. In Proceedings of the 3rd IEEE International Symposium on Dependable, Autonomic and Secure Computing (DASC’07), September 2007.

4