An ORB Endsystem Architecture for Statically ... - Semantic Scholar

An ORB Endsystem Architecture for Statically Scheduled Real-time Applications Douglas C. Schmidt, Rajeev Bector, David L. Levine, Sumedh Mungee, and Guru Parulkar fschmidt,rajeev,levine,sumedh,[email protected] Department of Computer Science, Washington University St. Louis, MO 63130, USA 

This paper will appear in the IEEE Workshop on Middle- it supports statically scheduled real-time applications. Our ware for Real-Time Systems and Services, San Francisco, CA, related work on CORBA explores other aspects of real-time ORB endsystem design, such as real-time request scheduling December 2nd, 1997. [5], real-time request demultiplexing [6], and real-time event service processing [7]. Abstract The paper is organized as follows: Section 2 describes the There is increasing demand to extend object-oriented middle- architectural components in TAO’s ORB endsystem that supware to support applications with stringent quality of service port statically scheduled applications with deterministic real(QoS) requirements. However, conventional object-oriented time requirements; Section 3 explains the enhancements we middleware does not define standard features for specifying or are making to the Solaris 2.x OS kernel to support end-to-end enforcing end-to-end QoS for applications with deterministic real-time functionality in TAO; and Section 4 presents conor statistical real-time requirements. This paper focuses on cluding remarks. strategies for integrating I/O subsystems with object-oriented middleware to produce ORB endsystems for statically sched2 Designing a Real-time ORB Endsysuled real-time applications.

tem

Keywords: Real-time CORBA, Object Request Broker, Quality of Service, OO Middleware, I/O Subsystem

2.1 Real-time Support in TAO To support the QoS demands of distributed object computing applications, an ORB endsystem must be flexible, efficient, Existing Object Request Broker (ORB) middleware like predictable, and convenient to program. To meet these requireCORBA [1], DCOM [2], and Java RMI [3] is suitable for re- ments, TAO provides end-to-end QoS specification and enquest/response applications with best-effort quality of service forcement mechanisms [5] and demultiplexing optimizations requirements. However, existing ORB middleware is not yet [6] for efficient and predictable request dispatching. The folsuited for performance-sensitive, distributed real-time applica- lowing overview of the real-time support in TAO’s ORB endtions due to (1) lack of QoS specification interfaces, (2) lack of system explains how its components can be configured to supQoS enforcement, (3) lack of real-time programming features, port statically scheduled real-time applications. and (4) lack of performance optimizations [4]. To address these shortcomings, we have developed The 2.1.1 Architectural Overview ACE ORB (TAO) [5]. TAO is a highly extensible, CORBA- Figure 1 illustrates the key components in TAO’s ORB endcompliant ORB endsystem targeted for applications with de- system. TAO’s I/O subsystem runs in the OS kernel and terministic and statistical quality of service requirements, as send/receives requests to/from clients across a network or well as best effort requirements. This paper addresses de- backplane. TAO’s ORB Core, Object Adapter, and servants scribes TAO’s ORB endsystem architecture and explains how run in user-space. TAO’s ORB Core contains a group of real-

1 Introduction

 This work was supported in part by Boeing, NSF grant NCR-9628218, DARPA contract 9701516, and US Sprint.

time threads that process client requests in accordance with their QoS requirements. TAO’s Object Adapter demultiplexes 1

Figure 2 illustrates an example of this type of application from the domain of avionics mission computing.

U

R U N T I M E S C H E D U L E R

CLIENTS

SERVANTS

STUBS

Q oS API

SKELETONS

SERVANT DEMUXER

OBJECT ADAPTER

ORB CORE REACTOR (20 HZ) HZ)

REACTOR (10 HZ) HZ)

REACTOR (5 HZ) HZ)

REACTOR (1 HZ) HZ)

I/O SOCKET QUEUE DEMUXER SUBSYSTEM

Z E R O C O P Y

HUD

Air Frame

Nav

3:PUSH (DEMARSHALED DATA) EVENT CHANNEL

B U F F E R S

2:PUSH (EVENTS) Sensor proxy

Sensor proxy

Sensor proxy

OBJECT REQUEST BROKER 1: SENSORS GENERATE

ATM PORT INTERFACE CONTROLLER (APIC)

DATA

Figure 1: TAO’s Real-time ORB Endsystem Architecture

Figure 2: Example Avionics Mission Computing Application Mission computing applications manage sensors and operator displays, navigate the aircraft’s course, and control weapon release. These applications require an ORB endsystem that supports real-time scheduling and dispatching of periodic processing operations, as well as efficient event filtering and correlation mechanisms [7]. To support these requirements, TAO’s ORB Core supports the rate-based concurrency model and rate-based connection model described below.

the requests using a servant demuxer, which associates requests with application-level servant operations in O(1) time. TAO’s servant demuxer uses perfect hashing and active demultiplexing [6] to dispatch client requests efficiently and predictably, regardless of the number of active connections, servants, and operations defined in IDL interfaces. One or more servants collaborate to implement application components or infrastructure components. Application components and infrastructure components are reusable building blocks that reside above an ORB. Typically, an infrastructure component represents an entity that is broadly reusable within a particular domain, such as real-time avionics or telecom call processing. In contrast, an application component captures the responsibilities of reusable entities within a particular project, such as different families of telecom switches. Complete applications can be configured by plugging application and infrastructure components together.

Rate-based concurrency model: TAO’s ORB Core can be configured to allocate a real-time thread for each rate group. A rate group assembles all periodic processing operations that occur at a particular rate. For instance, avionic mission computing systems commonly execute their tasks in rates groups such as 20 Hz, 10 Hz, 5 Hz, and 1 Hz. Each thread in TAO’s ORB Core can be configured with a Reactor that demultiplexes all incoming client requests for its rate group. The Reactor used in TAO is an objectoriented implementation of the Reactor pattern [8]. This pat2.1.2 Configuring TAO for Statically Scheduled Real- tern is used in TAO to demultiplex client requests to GIOP connection handlers. These handlers collaborate with TAO’s time Applications Object Adapter to dispatch requests to application-level serTAO can be configured to support different classes of appli- vant operations. TAO uses a Reactor, per thread, per rate group design to cation QoS requirements. The ORB endsystem configuration shown in Figure 1 is optimized for statically scheduled ap- minimize the number of threads in the ORB. The primary plications with deterministic, periodic real-time requirements. benefit of this design is its extremely low overhead for con2

text switching and synchronization. In particular, ORB and application component state need only be locked when objects in different rate groups interact. In addition, TAO ORB endsystem concurrency model supports scheduling and analysis techniques that associate priority with rate, such as Rate Monotonic Scheduling (RMS) and Rate Monotonic Analysis (RMA) [9, 10].

PERIODIC SCHEDULING AND DISPATCHING P TIME

C T

STUB

STUB

STUB

CLIENT ORB CORE CONNECTOR

CONNECTOR

CONNECTOR

20 10 5 1

20 10 5 1

20 10 5 1

HZ HZ HZ HZ

HZ HZ HZ HZ

HZ HZ HZ HZ

I/O SUBSYSTEM

struct RT_Info { Time_t worstcase_exec_time_; Time_t cached_exec_time_; Period_t period_; Importance importance_; sequence dependencies_; };

SERVER ORB CORE C O N N E C T 1

C O N N E C T 2

C O N N E C T 3

A C C E P T O R

REACTOR (20 HZ) HZ)

C O N N E C T 1

C O N N E C T 2

C O N N E C T 3

A C C E P T O R

REACTOR (10 HZ) HZ)

C O N N E C T 1

C O N N E C T 2

C O N N E C T 3

A C C E P T O R

REACTOR (5 HZ) HZ)

C O N N E C T 1

C O N N E C T 2

C O N N E C T 3

T+P T+2P P: PERIOD C: REQUEST COMPUTATION TIME

RT Operation

Rate-based connection model: Figure 3 illustrates TAO’s rate-based connection management model. Each client

CLIENT APPLICATION

C

A C C E P T O R

Figure 4: TAO’s QoS Specification Model

REACTOR (1 HZ) HZ)

2.2 Handling Real-time Client Requests in TAO

I/O SUBSYSTEM

Real-time applications that use TAO must specify their resource requirements to TAO’s Real-time Scheduling Service [5]. Currently, this QoS information is provided to TAO on a per-operation basis before program execution. For CPU requirements, the QoS requirements are expressed by RT Operations using the attributes of the RT Info IDL struct shown in Figure 4. An RT Operation is a scheduled operation, i.e., one that has expressed its scheduled resource requirements to TAO using an RT Info struct. RT Info attributes include worstcase execution time, period, importance, and data dependencies. Using scheduling techniques like RMS and analysis approaches like RMA, TAO’s Real-Time Scheduling Service determines if there is a feasible schedule based on knowledge of all RT Info data for all the RT Operations in an application. TAO’s QoS specification model is described further in [5]. Enforcement of requested resource requirements is not currently provided by TAO. Therefore, servants are responsible for ensuring that their usage does not exceed their specified QoS requests. We are adding enforcement in subsequent versions of TAO, using the admission control and enforcement mechanisms described in 3.3.

COMMUNICATION LINK

Figure 3: Inter-ORB Connection Topology thread maintains a Connector [11] in thread-specific storage. Each Connector manages a table of pre-established connections to servers. A separate connection is maintained for each rate. This design enables clients to preserve endto-end priorities while sending requests through ORB endsystems and across communication links. Also shown in in Figure 3 is how each rate group’s Reactor in a server is associated with an Acceptor [11]. The Acceptor is a factory that listens on a particular port number for clients to connect to that rate group. There is typically a listener port per rate, e.g., ports 10020, 10010, 10005, 10001.

Once a client connects, the Acceptor creates a new socket queue and GIOP connection handler to service that queue. The I/O subsystem uses the port number contained in arriving requests as a demultiplexing key to associate requests with the appropriate socket queue. This design minimizes priority inversion through the ORB endsystem by demultiplexing incoming requests onto the appropriate real-time thread that services 2.3 TAO’s Run-time Scheduling Model the rate group of the target servant. Section 3.5 explains how the early demultiplexing [12] feature of ATM can be used to The scheduling information used by TAO’s Run-time Schedvertically integrate TAO’s ORB endsystem from the network uler shown in Figure 1 consists of priorities and subpriorities. interface to the application servants. Applications can associate these with each servant operation, 3

as follows:

kernel level. Most control flows in the kernel, including interrupts, are threaded [13]. Priority: The priority is the (OS-dependent) thread priority The user-level programming model of Solaris supports mulof the highest priority thread than can execute the operation. It tiple threads of control within a single application process. is determined based on the maximum rate at which the servant Multiple application threads can be scheduled and run on could execute. For example, systems that are scheduled using one or more kernel threads (kthreads), which themselves are RMA can compute this information based on knowledge of the scheduled and run on one or more CPUs. Further details of computation time, Co , and period, Po , for each operation o. the Solaris multi-threading model are described in [13]. The Subpriority: The subpriority orders servant execution following discussion briefly outlines the Solaris scheduling within each rate group. This ordering is based on two factors: model and its protocol processing I/O subsystem. (1) data dependencies between servants and (2) the relative importance of servants. Data dependencies are expressed directly 3.1.1 The Solaris Scheduling Model to TAO’s Scheduler via the RT Info QoS specification mechanism described in [5]. In most cases, these dependencies can The scheduling support in Solaris is designed in a flexible manner. It is possible, for instance, to implement a customized be determined automatically. The subpriority of an operation also depends on its impor- scheduling policy as a scheduling class and load it into a runtance. Importance is an application-specific indication of rela- ning kernel. Table 3.1.1 illustrates the different scheduling tive operation importance. It is used by the Run-Time Sched- classes that are configured in the Solaris kernel by default. uler as a “tie-breaker” when other scheduling parameters, e.g., Interrupt (INTR) Used for servicing interrupts rate and data dependencies, are equal. Real-time (RT)

Fixed priority scheduling

The current TAO implementation supports off-line, static System (SYS) Only for system-specific threads Time-shared (TS) Time-shared scheduling scheduling. Its Run-Time Scheduler is therefore simply an interface to a table of scheduling information pre-computed off-line. When dynamic scheduling is supported by TAO, Table 1: Default Configured Scheduling Classes in Solaris (in the same interface will drive on-line algorithms and retrieve decreasing order of priority) scheduling output interactively. TAO’s Real-time Scheduling Service interface and its internal operation are described in [5]. The Solaris RT scheduling class attempts to provide worstcase guarantees on the time required to dispatch application or 3 Implementing the TAO ORB Endsys- kernel threads executing in this scheduling class [14]. Threads in the Solaris real-time scheduling class run at fixed priorities tem in Solaris that are always greater than those of threads in the time-shared and system scheduling classes. Only applications running with Meeting the QoS needs of distributed real-time applications superuser privileges can execute in the RT scheduling class requires more than defining IDL interfaces using OMG IDL since RT threads can take control of the system and starve out or developing an ORB Core with real-time threads. It requires all other system activities. Further information on the Solaris an OS I/O subsystem that supports real-time scheduling and scheduling classes appear in [14]. real-time communication. This section outlines the Solaris OS environment and ex3.1.2 The Solaris Protocol Processing I/O Subsystem plains why it is currently unsuitable for supporting a real-time ORB endsystem like TAO. Then, we explain the enhancements The Solaris communication I/O subsystem is based on the we are adding to the Solaris 2.x OS kernel to create an I/O STREAMS framework [15]. STREAMS provides protocol subsystem that supports end-to-end distributed real-time func- implementors with a variety of reusable components, such tionality for TAO. as buffer management, layered protocol module composition, and layer-to-layer flow control. In STREAMS, each protocol is implemented as a multiplexer or module and network drivers 3.1 Overview of Solaris implement the DLPI specification [16]. The Solaris kernel is a preemptive, multi-threaded, real-time, Protocol stacks written using Solaris STREAMS can be and dynamically loaded implementation of UNIX SVR4 and multi-threaded [17]. Multi-threading the protocol stack takes POSIX. It is designed to work on uniprocessors and shared advantage of multiprocessor machines that can execute mulmemory symmetric multiprocessors. Solaris contains a real- tiple connections or multiple packets in parallel, e.g., using time nucleus that supports multiple threads of control at the connection-oriented or message-oriented parallelism. 4

3.2 Limitations of Solaris Real-time Scheduling and Protocol Processing

Interrupt (INTR)

Interrupt thread executing protocols Preempts

High Priority CPU-bound thread

Although Solaris supports real-time CPU scheduling, it does not provide an integrated end-to-end solution for real-time ORB endsystems. The key limitations of the Solaris RT scheduling model are discussed below.

Real-Time (RT)

High Priority I/O-bound thread Low-Priority CPU-bound thread Depends on

System (SYS)

3.2.1 Lack of Integrated System-wide Real-time Scheduling

STREAMS thread at SYS priority

Time-Shared (TS)

The Solaris RT scheduling class only supports CPU-bound application threads. It is not integrated seamlessly with the Solaris STREAMS communication framework. In particular, the Solaris STREAMS implementation does not support QoS guarantees due to hidden scheduling and priority inversion. Hidden scheduling occurs when the kernel performs work asynchronously without regard to its priority. STREAMS processing in Solaris is a canonical example of hidden scheduling because the computation time is not accounted for by the application or the Solaris OS scheduler. To avoid hidden scheduling, the kernel should perform its work at the priority of the thread that requested the work. Priority inversion occurs when higher priority threads must block waiting for lower priority threads to release a resource required by the higher priority threads. Two common types of priority inversions in Solaris, thread-based and packet-based, are described below.

Figure 5: Common Priority Inversions in Solaris

Packet-based priority inversion: Even if thread-based priority inversion is bounded or eliminated, another potential priority inversion problem exists in Solaris STREAMS. This problem stems from the fact that many protocol implementations queue and process packets in FIFO order within STREAMS modules. FIFO queueing is prone to packet-based priority inversions. These inversions occur when higher priority threads must block until the packet they need to process is at the front of the queue. Both thread-based priority inversion and packet-based priority inversion are hard to solve generically because lower layers of the protocol stack are often unaware of the priorities of the packet’s sender/receiver thread. Section 3.4 outlines how we alleviate thread-based and packet-based priority inversion Thread-based priority inversion: Solaris uses a fixed in TAO’s I/O subsystem. global priority value for scheduling STREAMS threads. This priority lies in the range of priorities covered by the SYS scheduling class. Thus, STREAMS processing is done at a 3.2.2 Low-level QoS Support higher priority than all time-shared threads, but at a lower priThe Solaris real-time scheduling class is relatively low-level. ority than all real-time threads. For instance, there is no mechanism in Solaris for applicationEvery real-time application thread must depend on low prilevel QoS specification or enforcement, as described below: ority STREAMS kthreads to deliver network data before it can proceed. As a result, the Solaris I/O subsystem is prone to Lack of support for QoS specification: It is desirable to rethread-based priority inversions. These inversions occur when lieve applications from specifying low-level parameters, such a high-priority real-time I/O bound thread blocks indefinitely as thread or packet priorities. However, Solaris does not supwaiting for low-priority CPU-bound real-time threads to finport high-level QoS specifications. Ideally, an application ish. should only need to specify its QoS requirements at a highUnbounded thread-based priority inversion is highly undelevel, e.g., in terms of computation time or execution period. sirable in most real-time environments since it leads to nondeterministic behavior. In turn, this can result in missed dead- Lack of support for QoS enforcement: The RT scheduling lines for real-time application and ORB endsystem tasks. class in Solaris does not enforce QoS usage policies. ThereFigure 5 illustrates common sources of priority inversion fore, a CPU-bound RT program can take over the system and in Solaris. This figure also illustrates another limitation of starve all other system activities. Solaris regulates this undesirSolaris STREAMS for real-time ORB endsystems. Interrupt able situation by restricting the use of the RT scheduling class threads that execute protocol code can potentially preempt to applications running as superuser. However, this design is CPU-bound RT threads. This preemption is problematic if overly restrictive in a multi-user environment where multiple the data carried by the protocol packets is destined for lower- user-level applications may have QoS requirements. priority or non-real-time threads. 5

3.3 Overcoming Solaris Limitations for Dis- design allows the OS scheduler to either guarantee the resource availability specified by the application or refuse to adtributed Real-time Systems mit the thread. Admission control is useful for dynamically scheduled, deterministic and statistical real-time systems.

To overcome the limitations of Solaris discussed above, we are modifying the Solaris kernel to support real-time scheduling and protocol processing. These modifications provide the following facilities to give applications controlled access to OS resources:

Thread priority assignment: Once a thread of the RTIO class is accepted by the OS admission control mechanism, the real-time OS scheduler is responsible for (1) computing its priority relative to other threads in the class and (2) dispatching  CPU scheduling interface: This interface allows applica- the thread periodically so that its deadlines are met. tions to reserve a specified portion of CPU usage. Enforcement mechanisms: In many real-time applications,  Admission control: This mechanism in the kernel tracks threads that do not abide by their QoS specifications cannot system load and either admits the application requests or re- continue unabated. Thus, TAO’s I/O subsystem provides a jects it to ensure the system provides adequate QoS to appli- mechanism to enforce correct QoS behavior by applications. cations it has already admitted. It monitors the resource usage of the current thread and determines if it exceeds the usage that was guaranteed at the time  Enforcement mechanisms: These mechanisms monitor of admission. the system and ensure applications obey their QoS specificaQoS enforcement is not conceptually difficult to implement tions. since Solaris supports preemptive scheduling at the kernel level. Therefore, if a thread misbehaves, corrective or prevenThe remainder of this subsection describes the real-time I/O tive actions can be taken. Either the thread is denied service (RTIO) scheduling class we are adding to Solaris and explains or it can be requested to conform to its requested QoS via a how it enhances QoS specification and enforcement. In addi- callback mechanism [18]. tion, we outline how our I/O subsystem can be integrated with TAO’s ORB-level thread-per-rate concurrency model to alleviate scheduling hazards such as priority inversion and hidden 3.4 The RTIO Thread Pool Concurrency Model scheduling. TAO’s I/O subsystem uses a thread pool concurrency model. In this model, we bind a separate RTIO thread to each application thread, as shown in Figure 6. The RTIO thread pro-

3.3.1 The RTIO Scheduling Class We are enhancing the Solaris scheduling mechanism to support QoS guarantees by adding a new scheduling class that supports applications with periodic RTIO requirements. The RTIO scheduling class allows applications to specify their requirements in an intuitive manner. For instance, one implementation of the RTIO scheduling class is based on RMS. Using RMS, applications can specify their processing requirements in terms of computation time C and period P. The OS assigns priorities to RTIO threads so that schedulability is guaranteed. To provide QoS guarantees for protocol processing, we are integrating the Solaris STREAMS framework with the RTIO class. The global priorities of all threads in this RTIO class are higher than those of all other scheduling classes, except those of the Interrupt class. The modified Solaris real-time OS scheduler is responsible for the following operations:

INTR

Interrupt thread (Packet Classification) High Priority I/O-bound thread + STREAMS thread at same priority

RTIO

Low-Priority I/O bound thread + STREAMS thread at same priority

RT SYS TS

Figure 6: Alleviating Priority Inversion in TAO’s ORB Endsystem

Thread-level admission control: To ensure that a feasible schedule is produced, we are developing admission control strategies for the RTIO scheduling class. The admission control mechanism resides in the RTIO class and is involved whenever an application requests a certain QoS. This

cesses I/O requests in the STREAMS framework and the application thread processes client requests in the ORB. Both threads can have equivalent (high) priorities. Therefore, a 6

high-priority application thread will not starve while a lowpriority STREAMS thread performs protocol processing. This alleviates thread-based priority inversion for I/O bound tasks. In addition, we minimize the work done in the interrupt context, which alleviates hidden scheduling. Section 3.5 outlines the minimal functionality that must performed in the interrupt context. At ORB endsystem initialization time, a pool of kernel threads is allocated to perform subsequent protocol processing. The following are the main benefits of using thread pools:

executes the TCP/IP code and delivers the packet to the application

 Bounded load: The load on the system1 can be bounded by placing an upper limit on the percentage of CPU time that is available for RTIO threads.

1. Interrupt thread consults the Packet Classifier to enqueue the packet in the appropriate queue

3. The application consumes the data

APPLICATIONS

user kernel

2. RTIO thread

TCP/IP Code

Demultiplexing

ATM Driver

 Amortized thread creation cost: Thread creation overhead is not incurred at run-time because RTIO threads are pre-allocated at ORB endsystem initialization time. Each thread remains dormant until an application-level thread, e.g., in TAO’s ORB Core, is bound to it. At this point, the RTIO kthread starts executing protocol code on behalf of the application-level thread.

NETWORK Packet Classifier

Figure 7: Multi-threaded TCP/IP Processing with QoS

I/O threads in the Solaris kernel. To minimize priority inversion, the I/O threads perform protocol processing tasks for the corresponding Reactor threads. Figure 6 illustrates how thread-priority inversions are alleviated in TAO’s I/O subsystem by having a one-to-one binding between TAO user threads and STREAMS protocol kthreads and minimizing the work done in the interrupt context. Both the Reactor thread and its associated protocol thread are scheduled by the RTIO scheduling class in round-robin order. In this scheme, after either thread has a chance to run, the other thread must be scheduled. If the protocol thread has some packets for the application, the Reactor’s thread in the In a statically scheduled system, kthreads in the pool can ORB Core must run to consume the packets. Similarly if the be associated with different rate groups. This design comple- application has consumed or generated packets, the protocol ments the Reactor-based thread-per-rate concurrency model thread must run to deliver/send more packets. Protocol threads described in Section 2.1.2. Each kthread corresponds to a dif- run when the data is available and sleep otherwise. ferent rate of execution and hence will run at a different priority. Each thread in the pool is associated with a queue, as shown 3.5 Exploiting ATM Features to Minimize Packet-based Priority Inversion in Figure 7. The queue links the various protocol modules in a stream. Each thread has a rate assigned to it, which correATM is based on the concept of Virtual Circuits (VCs), which sponds to the frequency at which requests arrive from clients. switch ATM cells at high speeds. Each ATM connection has a Packets are placed in the queue by the application for the Virtual Circuit Identifier (VCI) associated with it. TAO’s I/O sender or by the interrupt handler for the receiver. Protocol subsystem associates thread priorities with VCIs and extends code is then executed by the thread to move the packet through the concept of a VC-based priority within the ORB endsystem the queue to the network interface card or to the application. through the ORB to the application servants. Our prototype implementation provides a one-to-one correTo distinguish packets on the basis of their priorities and spondence between Reactor threads in the ORB Core and classify them into appropriate queues, TAO’s I/O subsystem 1 Load is defined as the amount of CPU-time that is available to all threads exploits the early demultiplexing feature of ATM [12]. Early in the RTIO scheduling class. demuxing is the ability to detect the final destination of the

 Decoupled communication and computation: By using a pool of RTIO threads in the OS kernel, TAO’s ORB endsystem separates the application’s computation needs from its distributed communication needs. Thus, an application can specify one rate for protocol processing depending upon the network reserved rate and a different rate one for its own computation requirements. For instance, the rate at which the network delivers packets to the end-system will determine how much CPU time should be allocated to the thread that performs protocol processing.

7

packets based on the VCI field in the packet. TAO uses the early demultiplexing feature of ATM to vertically integrate its ORB endsystem architecture, as described in Section 2.1.2. The use of early demultiplexing also helps to alleviate packet-based priority inversion because packets need not be queued in FIFO order. Instead, TAO’s I/O subsystem supports priority-based queuing, where packets destined for higherpriority applications are delivered ahead of lower-priority packets that remain unprocessed in their queues. In contrast, conventional implementations of network protocols in Solaris process all packets at the same priority, regardless of the application for which they are destined. At the lowest level of TAO’s ORB endsystem, the ATM driver distinguishes between packets based on their VCIs and enqueues them in the appropriate STREAM queues. The RTIO kthread associated with the STREAM delivers the packets to the ORB Core, as shown in Figure 9. Our design minimizes the amount of processing done at interrupt level because Solaris does not consider packet or RT thread priority when invoking interrupt routines. Protocol processing in the TAO ORB endsystem can be done per STREAM and hence at different priorities, rather than being done globally for the entire system. This design helps alleviate priority inversion when many processes running at different priorities are engaged in network I/O. Figure 8 shows how the packet classification mechanism works for the receive-side of TAO’s I/O subsystem.

Priority1

Priority 3

Priority 4

should be placed. This is required when multiple applications use a single VC or when the link layer is not ATM. In these cases, it is necessary to identify packets and associate them with rates/priorities on the basis of higher-level protocol addresses like TCP port numbers. The device driver can be modified to search the Run-Time Scheduler table in the ORB’s memory. TAO’s Run-time Scheduler maps TCP port numbers to rate groups in O(1) time.

3.6 Summary of Components in TAO’s I/O Subsystem Figure 9 illustrates the key components in Solaris that we have extended and integrated into TAO’s real-time I/O subsystem. These components consist of the following: PERIODIC APPLICATIONS

APERIODIC APPLICATIONS

C

BEST EFFORT APPLICATIONS

C

t t D T T: period D: deadline C: computation time C: computation time

PACKET PROCESSING THREADS (RTIO Class)

SCHEDULER AND ADMISSION CONTROL

APP

APP

APP

APPS

ORB

ORB

ORB

TAO ORB CORE

RIOP

RIOP

TCP/IP

PROTOCOL PROCESSING MODULES

REAL-TIME THREADS

VC2 VC3 111111 000000 00 11 000000 111111 00 11 0000 1111 DEMULTIPLEXING 000000 111111 00 11 0000 000000 111111 001111 11 0000 1111 VC1

DEVICE DRIVER AND PACKET CLASSIFIER

AAL FRAMES

ATM DRIVER 3. ENQUEUE

00000000000000000 11111111111111111 00000000000000000 0000011111111111111111 11111 1111 000000000000000000000 11111111111111111

NETWORK

PACKET CLASSIFIER 2. LOOK-UP 1. INTERRUPT

DEVICE QUEUE

VCI

QueueID(ptr)

1

3232323

2

3435345

4

3232355

Figure 9: Vertically Integrated Real-time Subsystem

 STREAMS implementation: The TCP/IP and AAL-5 layers in the Solaris STREAMS framework are modified to allocate multiple queues when they are initialized. In addition, the queueing procedures in the Solaris STREAMS implementation are extended to handle multiple queues and link them together so that the packet is passed up in the correct queues. Figure 7 illustrates how the packet priorities are maintained until the packets are delivered to the application.

1111111111111111 0000000000000000 0000000000000000 NETWORK 1111111111111111 Figure 8: Packet Classifier

 VCI to STREAMS mapping: The kernel stores the mapping of VCIs to STREAMS queues. This mapping is used by

The Packet Classifier can be configured to consult TAO’s real-time Scheduling Service to determine where the packet 8

the AAL-5 layer to demultiplex and insert the packets into the appropriate queues.

CORBA,” IEEE Communications Magazine, vol. 14, February 1997.

 RTIO scheduling class: This scheduling class is responsible for admission control and periodic activity of the threads belonging to this class. Threads in the I/O subsystem, ORB, and application components can all run in the RTIO scheduling class.

[5] D. C. Schmidt, D. L. Levine, and S. Mungee, “The Design and Performance of Real-Time Object Request Brokers,” Computer Communications, to appear, 1998. [6] A. Gokhale and D. C. Schmidt, “Evaluating the Performance of Demultiplexing Strategies for Real-time CORBA,” in Proceedings of GLOBECOM ’97, (Phoenix, AZ), IEEE, November 1997.

 RTIO threads: Kernel threads running in the RTIO scheduling class will store the index into the STREAM queue that they intend to process. All protocol processing is performed in the context of these threads.

[7] T. H. Harrison, D. L. Levine, and D. C. Schmidt, “The Design and Performance of a Real-time CORBA Event Service,” in Proceedings of OOPSLA ’97, (Atlanta, GA), ACM, October 1997. [8] D. C. Schmidt, “Reactor: An Object Behavioral Pattern for Concurrent Event Demultiplexing and Event Handler Dispatching,” in Pattern Languages of Program Design (J. O. Coplien and D. C. Schmidt, eds.), pp. 529–545, Reading, MA: AddisonWesley, 1995.

4 Concluding Remarks

Conventional operating systems and ORBs do not provide adequate support for the QoS requirements of distributed, real[9] C. Liu and J. Layland, “Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment,” JACM, vol. 20, time applications. Meeting these needs requires an integrated pp. 46–61, January 1973. ORB endsystem architecture that delivers end-to-end QoS guarantees at multiple levels of the entire system. The TAO [10] M. H. Klein, T. Ralya, B. Pollak, R. Obenza, and M. G. Harbour, A Practitioner’s Handbook for Real-Time Analysis: Guide ORB endsystem architecture described in this paper addresses to Rate Monotonic Analysis for Real-Time Systems. Norwell, this need by combining a real-time I/O subsystem with a careMassachusetts: Kluwer Academic Publishers, 1993. fully designed ORB Core and Object Adapter. [11] D. C. Schmidt, “Acceptor and Connector: Design Patterns for The initial focus of the TAO project has been on statiInitializing Communication Services,” in Pattern Languages of cally scheduled applications with deterministic QoS requireProgram Design (R. Martin, F. Buschmann, and D. Riehle, ments. We are currently extending TAO to support dynamieds.), Reading, MA: Addison-Wesley, 1997. cally scheduling and applications with statistical QoS require- [12] Z. D. Dittia, G. M. Parulkar, and J. Jerome R. Cox, “The APIC ments. The source code for ACE and TAO is freely available Approach to High Performance Network Interface Design: Protected DMA and Other Techniques,” in Proceedings of INFOat www.cs.wustl.edu/schmidt/TAO.html. COM ’97, (Kobe, Japan), IEEE, April 1997. [13] J. Eykholt, S. Kleiman, S. Barton, R. Faulkner, A. Shivalingiah, M. Smith, D. Stein, J. Voll, M. Weeks, and D. Williams, “Beyond Multiprocessing... Multithreading the SunOS Kernel,” in Proceedings of the Summer USENIX Conference, (San Antonio, Texas), June 1992.

Acknowledgements

We gratefully acknowledge the support and direction of the Boeing Principal Investigator, Bryan Doerr. In addition, we thank Chris Gill for valuable comments on this paper, [14] S. Khanna and et. al., “Realtime Scheduling in SunOS 5.0,” in Proceedings of the USENIX Winter Conference, pp. 375–390, Chris Cleeland for implementing TAO’s ORB Core, and Andy USENIX Association, 1992. Gokhale for implementing the GIOP protocol engine, request [15] D. Ritchie, “A Stream Input–Output System,” AT&T Bell Labs demultiplexer, and IDL compiler. Technical Journal, vol. 63, pp. 311–324, Oct. 1984. [16] OSI Special Interest Group, Data Link Provider Interface Specification, August 1991.

References

[17] S. Saxena, J. K. Peacock, F. Yang, V. Verma, and M. Krishnan, “Pitfalls in Multithreading SVR4 STREAMS and other Weightless Processes,” in Proceedings of the Winter USENIX Conference, (San Diego, CA), pp. 85–106, Jan. 1993.

[1] Object Management Group, The Common Object Request Broker: Architecture and Specification, 2.0 ed., July 1995. [2] D. Box, Understanding COM. Addison-Wesley, Reading, MA, 1997.

[18] J. A. Zinky, D. E. Bakken, and R. Schantz, “Architectural Support for Quality of Service for CORBA Objects,” Theory and Practice of Object Systems, vol. 3, no. 1, 1997.

[3] A. Wollrath, R. Riggs, and J. Waldo, “A Distributed Object Model for the Java System,” USENIX Computing Systems, vol. 9, November/December 1996. [4] D. C. Schmidt, A. Gokhale, T. Harrison, and G. Parulkar, “A High-Performance Endsystem Architecture for Real-time

9