checkers like AIDE or Tripwire. Flight Data Recorder. Event logging by system loggers is usually used for post-mortem analysis of system crashes. However, the ...
Defending Network-Centric Systems using Backdoors Liviu Iftode, Arati Baliga, Aniruddha Bohra, Stephen Smaldone, and Florin Sultan Department of Computer Science Rutgers University, Piscataway, NJ 08854-8019, U.S.A. iftode, aratib, bohra, smaldone, sultan
cs.rutgers.edu
As computing systems are increasingly depending on networking, they are also becoming more vulnerable to networking malfunctioning or misuse. Human intervention is not a solution when computer system monitoring and repairing must be done fast and reliably regardless of scale, networking availability, or system impairing. Future network-centric systems must be built around a defensive architecture that allows computers to take care of themselves. In this paper, we argue that the solution to building self-defending computer architectures is a Backdoor, which can support automated observation and intervention on a computer system’s memory without involving its operating system. Backdoors can therefore execute even when the functionality of the operating system of a critical system has been severely compromised and the system is no longer accessible through the primary network. Backdoors can be realized in hardware over a programmable network interface or in software over a virtual machine monitor.
I.
Motivation and Goals
Despite decades of work on verification techniques, fault tolerance and security, computers and networks continue to remain vulnerable to failures and attacks. Increasing system complexity certainly does not help, even worse, it makes humanassisted monitoring, maintenance and intervention not only prohibitively costly but, more dramatically, unacceptably slow and sometimes ineffective. This problem is only going to become worse in the future, as the number of computing devices per person will increase while the vision of pervasive computing approaches its realization. In response to these trends, research focus has recently started to shift from eliminating failures towards designing systems that can withstand them and recover fast.1 What makes computers ultimately and persistently vulnerable? We believe that the ultimate cause of computer vulnerability is the computer architecture itself. Placing trust in the operating system is not a guarantee for dependability as long as the operating system executes from the main memory, which can be accidentally or intentionally violated. Moreover, although operating systems have been extensively crafted to be safe, holes continue to exist and will always exist. A common source of failures, operating system extensibility, has been recently addressed by projects like Nooks.2 More recent research has suggested to move the trust from software to hardware.3 The problem with this approach is that it requires a modified processor architecture, not trivial to deploy across the huge computing infrastructure operational today. The question is whether we can improve computer and network dependability in the presence of failures and attacks by strengthening rather than replacing the existing architectural framework. A recent example of technological prowess, the remote repair of one of the Mars rover computers, holds, in our opinion, the answer. In the Mars rover case, a computer hung and then entered a pattern of periodic reboots because it ran out of flash memory due to an excessive, unanticipated, accumulation of logged data. What made repair possible was a provision that allowed remapping of the memory used during reboot from flash memory to RAM, creating an alternate execution path that bypassed the faulty computer resource.4 This example suggests building systems capable of automated monitoring and intervention when their main functionality is compromised by faults or attacks. However, unlike the rover computer, the solution must be cost-effective and easy to deploy on the existing computing platforms, by using only conventional processor, system, and OS architectures. In this paper, we argue for augmenting existing computer systems and networks with trusted, multi-layered “last lines of defense.” A last line of defense provides a last resort for access to and action on resources of a computer even when This work is supported in part by NSF CAREER Award CCR 0133366.
1 of 7 American Institute of Aeronautics and Astronautics
processors or I/O devices are not available due to failures or attacks. We describe our vision of defensive architectures built around the existing computing infrastructure. The basic building block of Defensive Architectures (DAs) is the Backdoor (BD), a trusted intelligent device that sits on the I/O bus of the system and can connect to other similar devices through a local fabric or a secure private network. Backdoors can be programmed to execute various defensive activities such as monitoring, logging, diagnosis and healing, using local and remote access operations to memory and I/O devices. To be secure, a backdoor should not be observable or controllable by the local processor or by any single remote backdoor. In particular, it must be resistant to tampering by malicious or faulty software running on its host computer. In DA, the backdoors will be programmed to execute defensive activities to protect computers and networks of computers in three defensive architectures: (i) Defensive Computer Architecture (DCA) - a conventional computer architecture augmented with a backdoor interface; (ii) Defensive Network Architecture (DNA) - a local network of backdoors, each of them attached to a computer; (iii) Defensive Inter-Networking Architecture (DINA) - a private network of DNAs, potentially distributed over the wide area. Achieving these goals requires solutions to two significant research challenges. First, we must be able to build backdoor capabilities into an existing intelligent network interface (I-NIC). In addition to local and remote memory read/write operations, the I-NIC must be capable of (i) initiating I/O operations with other local or remote I/O devices without involving the host processors, and (ii) preventing local processors or other I/O devices from turning off or compromising BD functionality. Second, we must design a software architecture that can be programmed to execute a wide range of defensive actions within the existing hardware and software platforms. We believe that DA will radically change the way we design survivable systems, leading to a new area of research with a rich problem space. At the same time, a Defensive Architectures implementation, possible using off-the-shelf technology, makes it immediately applicable to real-world systems resilient to failures and attacks. This will address the dependability problem and reduce the human cost that is now confronting owners and system administrators who have to monitor, configure, and maintain large-scale systems.
II. The Backdoor A. Backdoor Architecture The Backdoor architecture is a combination of hardware and low-level software (firmware). The BD hardware is essentially a programmable network interface (I-NIC) (a NIC equipped with its own processor and memory) that (i) sits on the I/O bus of the computer, and (ii) can connect to backdoors of other computers through a trusted, secure network fabric/backplane. Placing processing power on a NIC has long been used in commercial implementations to increase communication performance by implementing low-latency, high-bandwidth communication models,5, 6 accelerating and/or offloading execution of IP stack protocols,7, 8 offloading remote storage protocols,6, 7, 9 etc. While in these examples some functionality is provided by a NIC, they differ in their flexibility of programming. In BD, we specifically rely on the programmability of the NIC to implement access operations on the resources of its host computer. Figure 1 depicts the software BD architecture. The “door” in the top left corner represents a (logical) access port controlled by the BD firmware. The door status (open/closed) indicates whether the BD accepts accesses to its resources initiated by the local processors or other I/O devices. When a BD is powered on, the door is open. This allows the local OS to program the BD to perform the desired operations. During the normal operation (after initialization) the BD can leave the door open or it can close it. Once the door is closed (at the end of the initialization phase), it cannot be opened again by any action performed by the local processors or I/O devices. Accesses to local memory or I/O devices initiated by the BD are always possible, regardless of the status of the door. The BD software architecture consists of the following components: (i) an operation queue where BD operations sent by the local processors and/or by the remote BDs are stored, (ii) an operation scheduler that dispatches queued operations repeatedly, according to their specified frequency, (iii) a basic engine that executes local or remote operations, (iv) an inter-BD protocol module, and (v) specialized functional units implementing complex actions (e.g., operation agreement for validating remote operations). Remote operations are forwarded to the BD of a corresponding peer. The inter-BD protocol must include the door status with each remote request. On receiving an operation from a remote peer, a BD can decide to service it right away (if the door status is closed) or validate it before servicing it (if the door status is open). This is necessary because an operation received
2 of 7 American Institute of Aeronautics and Astronautics
Figure 1. The Backdoor software architecture.
from an open backdoor could be initiated or corrupted by its host OS, which the BD may not trust. To handle this case, the BD must be programmed to validate “insecure” remote operations through an agreement scheme that requires a select set of backdoors to send the same operation. To implement the backdoor functionality, the I-NIC must support the following operations: 1. Access local and remote memory in cooperation with a remote BD without involving the remote CPU. This capability is present in implementations of remote memory communication in commercial I-NICs.5 2. Access local and remote I/O devices without involving local/remote CPU. This capability is supported by device-todevice communication over the I/O bus, present in switched-based I/O interconnects. 10 3. Control accesses to local resources such as the BD control registers and memory. This capability is to be soon incorporated in commercial NICs, e.g., Myrinet.11 B. Backdoor Programmability To program a BD for defensive activities, a sequence of operations generated by the local host is loaded into the BD operation queue. For convenience, we call this sequence a BD program, not to be confused with the immutable and protected firmware controlling the BD. Operations in a BD program are tuples (written in a simple scripting language) that define rules of the form “if then ”, to be executed periodically by the execution engine of the BD. These rules are “programmed” into the BD by the local host during its safe initialization, or by remote BDs through the inter-BD protocol (without involving local processors). Note that programming a rule does not mean to download code into the BD. Rather, rules are expressed as parameterized “macrooperations” already implemented in BD functional units. Rules may contain symbolic references to kernel variables which are translated into physical references when the program is loaded. Programming the BD can be done by the OS through a special BD driver that has only an initialization routine. Its sole purpose is to load the BD operation queue once, at system initialization time, through a secure and protected sequence that does not access any other BD resource. For example: (i) the driver issues commands to the BD to load a program into its operation queue; (ii) when done, the driver signals the end of the sequence; (iii) in response, the BD closes the door to prevent any further accesses and start execution of the program. A challenging task is how to guarantee the integrity of the BD program, depending on the degree of trust the BD can place in the OS during initialization. In a trusted environment, running the driver by the OS in single-user mode would suffice to prevent corruption of the loaded BD program. However, in the extreme case when the integrity of the OS, of the driver and/or of the program are not to be trusted, cryptographic techniques may be required to validate the BD program. Possible solutions rely on the ability to store immutable cryptographic hashes of the valid programs in the NIC only once, in a secure environment. This is acceptable, given the BD program does not change unless changes are made to the hardware or software configuration and the defensive activities must be reprogrammed. The BD will then check the integrity of the program on every (untrusted) load by the driver, by recomputing and validating the hash over the loaded program.
3 of 7 American Institute of Aeronautics and Astronautics
III.
Defensive Architectures
Figure 2. The Defensive Computing Architecture augments an existing computer system with a trusted backdoor.
The backdoor is a basic building block in three Defensive Architectures. Defensive Computer Architecture (DCA). The DCA (Figure 2) is essentially a computer system, augmented with a trusted backdoor placed on the system I/O bus. DCA logically partitions a computer system into “frontdoor” components (processors, memory, I/O controllers, etc.) and its “backdoor” component. Frontdoor components, which provide the base functionality of the system, are usually exposed to interactions with the outside world (e.g., through networking) and are under the control of the OS and application software. This makes them and the functions they provide vulnerable to failures, attacks, software crashes, human errors, etc. In contrast, the backdoor runs a trusted and protected firmware that controls both the backdoor interactions with the host and with other backdoors. Defensive Network Architecture (DNA). The DNA (Figure 3) is a localized cluster of DCA nodes whose BDs are connected over a high bandwidth, low latency interconnection fabric. For a BD to be part of the DNA, it must (i) allow protected remote access to resources of its host (memory, I/O devices, etc.), and (ii) adhere to the inter-BD protocols to authenticate and validate requests. Although DNA is intended to be under the complete control of secured BDs, it is possible to design a DNA where certain or all hosts are involved in the defense. A host can provide, for instance, additional resources (persistent storage, large memory, and fast processor) for monitoring and integrity constraint validation. When hosts are involved in the defense, the door between them and the local BD might be left open after initialization. In this case, the BDs can be programmed to perform agreement on operations and additional integrity checking.
Figure 3. The Defensive Network Architecture connects BDs of computers in a LAN-based cluster through a specialized interconnect.
Defensive Inter-Network Architecture (DINA). The DINA (Figure 4) is an internetwork of DNA clusters and trusted hosts which are potentially geographically distributed. In DINA, the backdoors of each DNA cluster connect through a DNA Gateway to a secure private inter-cluster network. The DNA Gateway is a trusted system, i.e., which is not connected to the WAN or other I/O devices through which attacks or faults can be generated. The DNA Gateway plays two roles: (i) tunnels BD operations from one DNA cluster to another over the private network, and (ii) executes meta-operations such as BD operation multicast. 4 of 7 American Institute of Aeronautics and Astronautics
Figure 4. The Defensive Inter-Network Architecture uses DNA gateways interconnected on a private network to access BDs in nodes/clusters dispersed over wide-area.
IV. Defensive Activities We describe several examples of Defensive Activities that can be implemented as BD programs. A. Applications of the DCA Smart Watchdog. A local BD can be programmed to monitor OS invariants in the host memory or the I/O system and take immediate actions when they are violated. Invariants can be as simple as statistics on a certain OS event, such as the frequency of an interrupt, or as complicated as a file system consistency test. Actions may include turning off the interrupt bit for a malfunctioning device, overwriting a memory region to refresh software state and/or clean-up corrupted state of an OS subsystem, halting processors or I/O devices, and, most drastically, rebooting the system. The idea of using dedicated hardware which is not under the control of the host OS for monitoring or to augment the functionality of a system has been used in several previous projects. Custom hardware for sensing, diagnosing and controlling computer hardware components in a large-scale cluster has been proposed in.12 Self-securing intelligent devices are proposed in13 to provide better system security through protected access control. A programmable secure coprocessor that provides highly specialized support for complex secure applications has been developed by IBM.14 Finally, 15 describes a file integrity verification system based on an independent auditor implemented as an intelligent PCI card that can securely execute integrity checkers like AIDE or Tripwire. Flight Data Recorder. Event logging by system loggers is usually used for post-mortem analysis of system crashes. However, the tail of the log, which holds the richest information about the succession of events that led to the crash, is often lost because it is kept in system memory. To prevent loss of the event log tail, BD can act as a “flight data recorder” that keeps a short log over a window of most recent events. Moreover, BD can be programmed to dump logs on a local disk by executing direct BD-to-disk DMA transfers on the I/O bus. Buffer Cache Synchronization. Buffer cache synchronization in the event of a crash can be implemented as a BD program that detects the crash (e.g., by monitoring system activity) and writes-back the buffer cache to disk to save unsync’ed data. To prevent the machine from being rebooted while syncing to disk is still in progress, the BD can either block or intercept the host reboot sequence and stall it by synchronizing with the processor commanding the reboot. B. Applications of the DNA OS Integrity Verification. An OS subject to failure or attack cannot be fully trusted with accurate self-monitoring and self-checking. DNA solves this problem by performing integrity checks through the secure path established between peer BDs. The BD can verify the integrity of the OS running on its host by periodic comparisons with reference copies running or stored on other machines. On discovering a violation, the BD can take actions to (i) protect other hosts from a corrupted or compromised host by blocking its access to the DNA network, (ii) refresh/patch the OS state, halt or reboot. Remote Logging. Logging the system activity in an OS relies on resources of the system for initiating and recording the logs. This limits the usefulness of logging when scarcity of resources (e.g., due to CPU, memory or disk contention) delays or prevents logged information from reaching its consumers. Worse, an attacker/intruder that takes control of a system can 5 of 7 American Institute of Aeronautics and Astronautics
simply wipe out the logs to hide illicit activities. DNA solves this problem by remote logging, i.e., storing the logs in the remote memory or on remote disks. The BD can retrieve the OS-produced logs from system memory (where they are stored before being flushed to disk) and send them to another logger BD that stores them. Fast Reboot and State Refreshing. Rebooting is often performed to bring a system to a clean state. Its high downtime cost (reading the OS image from disk, wasted useful state, consistency checks, application restart, etc.) is often paid just to re-initialize a single component subsystem.16 DNA can support fast reboot by saving in-memory state of the critical OS data structures in the memory of other nodes, and restoring it during reboot. The buffer cache can be made persistent across reboots, without using non-volatile memory and involving the processors or the OS in saving/restoring the state. DNA also enables fast refresh of OS subsystem state, without a full system restart, using consistent state snapshots periodically taken by the local BD. C. Applications of the DINA A News Agency. DINA can support a global secure information network, subscribed to by critical system controllers (routers, GRID control nodes, PlanetLab peers, etc.) interested in information about the Internet, individual networks, or hosts. DNA clusters produce information that is summarized and disseminated by their DNA Gateways, to be consumed by other DNA clusters or controller systems. Interested controllers can then request additional, fine-grained information from DNA Gateways about their clusters and use it to diagnose problems and initiate repair. Early Warning System. The spreading rate of Internet worms is too high for any attempt by human operators to protect networks against them: information and awareness come too late to help effective action. DINA offers a solution by propagating information generated by the DCA systems (integrity reports, logs, etc.), through the DNA Gateway, to specialized hosts that analyze it, identify threats, warn and protect other clusters against similar attack. For proprietary systems, e.g., multiple instances of an Internet service, the system can be also used to monitor cluster performance, e.g., to detect a load spike. The controller services, e.g., a CDN, can globally identify hot-spots and balance the load among its clusters. Recover Critical State. Long-running, performance-critical Internet systems (e.g., content distribution servers, caching proxies, routers, etc.), build up their state over months of operation. This state, albeit soft, is critical for the service they provide, therefore sensitive to failures. Saving it to disk for persistence hurts system performance. DINA can solve this problem by nonintrusive state storage and retrieval on remote nodes.
V. Preliminary Results We have implemented a BD prototype using Myrinet NICs5 for remote memory access, and used it for remote healing within a cluster of Internet servers running RUBiS (a complex, multi-tier, transactional auction service similar to e-Bay). Preliminary results17–19 show that nonintrusive remote access to the memory of a computer system can achieve accurate monitoring and efficient repair/recovery actions without using processors or OS resources of a failed system. Under a moderate workload with 200 clients, the system resumed service to the clients in under two seconds (quiescent time mainly due to cluster reconfiguration), without losing any requests, while maintaining exactly-once execution semantics across crash.
VI.
Virtual Backdoor
In previous sections, we describe a Backdoor instantiated over a programmable Intelligent Network Interface Card (INIC) to monitor, recover, and repair systems from software faults and attacks. While it is easily deployable, built out of commodity components, and does not require any modification to the OS or applications, it requires additional hardware. As an alternative, the Backdoor can be realized over Virtual Machine Monitors (VMMs).20, 21 VMMs are implemented in software and execute at the highest privilege level in the system. The OS and applications are executed on top of these VMMs at a lower privilege and cannot directly access hardware. Instead, all hardware access is controlled through the VMM. Apart from providing performance isolation, VMMs have been previously used for intrusion detection, software debugging, and fault isolation.22, 23 A virtual backdoor can be implemented as a software component in a priviledged domain executing over the VMM, with hooks to execute in the privileged VMM domain. A specialized application running inside this priviledged domain can use the virtual backdoor to access the Guest OS and application state. The design principles of the virtual backdoor are similar to
6 of 7 American Institute of Aeronautics and Astronautics
those described previously. While using a virtual BD enables easier development and deployment of the defensive activities, it relies on the correct execution and integrity of the VMM in addition to the BD programs that have been deployed on it. However, several commercial and academic research projects have demonstrated that highly reliable VMMs can be built without compromising performance. We believe, with the growth of the VMM technology and the hardware capabilities, virtual BD would be an important component for monitoring, recovery, and repair of the system. We are currently developing an implementation of a virtual Backdoor, as an enhancement to an existing VMM.
References Patterson, D. et al., “Recovery Oriented Computing (ROC): Motivation, Definition, Techniques, and Case Studies,” Tech. Rep. UCB//CSD-02-1175, UC Berkeley Computer Science, March 2002. Swift, M. M., Bershad, B. N., and Levy, H. M., “Improving the Reliability of Commodity Operating Systems,” Proc. 19th Symp. on Operating Systems Principles (SOSP), Oct. 2003. Lie, D. et al., “Architectural Support for Copy and Tamper Resistant Software,” Proc. 9th ASPLOS, 2000. “NASA JPL, Jan. 24-26, 2004 Press Releases,” http://marsrovers.jpl.nasa.gov/newsroom/pressreleases. “Myricom: Creators of Myrinet,” http://www.myri.com. “Emulex, Inc.” http://www.emulex.com. “Alacritech Storage and Network Acceleration,” http://www.alacritech.com. “Intel(R) IXP Network Processor Family,” http://www. intel. com/design/network/products/npfamily. “Cyclone Intelligent I/O,” http://www.cyclone.com. “PCI-SIG - PCI Express,” http://www.pcisig.com. Seitz, C. L., “Personal communication,” Feb. 2004. Oppenheimer, D., Oppenheimer, D., Brown, A., Beck, J., ant J. Kuroda, D. H., Treuhaft, N., Patterson, D., and Yelick, K., “ROC-1: Hardware Support for Recovery-Oriented Computing,” IEEE Trans. Comput., Vol. 51, No. 2, 2002, pp. 100–107. Ganger, G. R. and Nagle, D. F., “Better Security via Smarter Devices,” Proc. HotOS VIII Workshop on Hot Topics in Operating Systems, 2001. Dyer, J. G., Lindemann, M., Perez, R., Sailer, R., van Doorn, L., Smith, S. W., and Weingart, S., “Building the IBM 4758 Secure Coprocessor,” Computer, Vol. 34, No. 10, Oct. 2001, pp. 57–66. Molina, J. and Arbaugh, W. A., “Using Independent Auditors as Intrusion Detection Systems,” 4th Information and Communications Security International Conference, Dec. 2002. Candea, G. and Fox, A., “Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel,” Proc. HotOS VIII Workshop on Hot Topics in Operating Systems, 2001. Sultan, F., Bohra, A., Neamtiu, I., and Iftode, L., “Nonintrusive Remote Healing Using Backdoors,” In Proc. First Workshop on Algorithms and Architectures for Self-Managing Systems, Self-Manage 2003, ACM, San Diego, June 2003. Sultan, F., Bohra, A., Gallard, P., Neamtiu, I., Smaldone, S., Pan, Y., Neamtiu, I., and Iftode., L., “Recovering Internet Service Sessions from Operating System Failures,” IEEE Internet Computing, Special Issue-Recovery Oriented Approaches to Dependability, Vol. 9, No. 2, March-April 2005, pp. 17–25. Bohra, A., Neamtiu, I., Gallard, P., Sultan, F., and Iftode, L., “Remote Repair of OS State Using Backdoors,” Proc. Int’l. Conference on Autonomic Computing, May 2004. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A., “Xen and the Art of Virtualization,” SOSP ’03: Proceedings of the nineteenth ACM symposium on Operating systems principles, ACM Press, New York, NY, USA, 2003, pp. 164–177. Waldspurger, C. A., “Memory Resource Management in VMware ESX Server.” In Proc. Symposium on Operating System Design and Implementation, OSDI’02, 2002. King, S. T. and Chen, P. M., “Backtracking intrusions,” ACM Trans. Comput. Syst., Vol. 23, No. 1, 2005, pp. 51–76. Garfinkel, T., Pfaff, B., Chow, J., Rosenblum, M., and Boneh, D., “Terra: a virtual machine-based platform for trusted computing,” SOSP ’03: Proceedings of the nineteenth ACM symposium on Operating systems principles, ACM Press, New York, NY, USA, 2003, pp. 193–206.
