user level thread is supported by one or more virtual processors (VPs), i.e. kernel level threads. ... In addition, abstract resources are managed by managers.
Reconfiguration of Resources in Middleware Hector A. Duran-Limon, Gordon S. Blair Computing Department, Lancaster University, Bailrigg, Lancaster LA1 4YR, UK. Tel: +44 1524 593141 Fax: +44 1524 593608 {duranlim, gordon}@comp.lancs.ac.uk
Abstract: The monolithic and inflexible nature of current middleware has made it difficult to deal with emerging technologies such as multimedia. Since these applications have timeliness constraints, they are highly susceptible to dynamic and unexpected changes. Nowadays, the attempts to introduce more flexibility in middleware have not been addressed in a generic and principled way. We believe that reflection provides a principled means to achieve the flexibility and adaptation required. The main focus of this paper regards the reconfiguration of resources in middleware within the context of OpenORB, a reflective middleware architecture. A resource framework is presented which encompasses both a resource model and a task model. The resource management of the logical flow of component interactions is modeled within the task model. The resource model is then used to provide a representation of the physical resources whereby various levels of abstraction are offered. The conjunction of both models offer a high-level of resource management. The approach is validated by a series of experimental results.
1 Introduction The last few years have seen a rapid expansion of computer network systems accompanied with the development of different operating systems and hardware platforms. This resulted in problems of heterogeneity and complexity. To tackle this situation, middleware technologies, such as CORBA and DCOM, have recently been developed. These technologies have succeeded in providing standards for a common programming model that addresses the problems mentioned above. However, the monolithic and inflexible nature of current middleware has made it difficult to deal with emerging technologies such as multimedia. That is, distributed multimedia applications are greatly affected by changes introduced to their environment due to the timeliness constraints imposed these systems. Therefore, there is a need to achieve support for both configuration and reconfiguration in middleware. The former is required in order to meet the specific requirements of the platform of deployment. The latter is then needed to adapt the system when unexpected changes occur. We believe that resource management plays an important role in this adaptation process. There has been some work on this area [OMG99a, Pal00, Schmidt97], however, we believe that the resource configuration facilities provided by these solutions are not comprehensive enough. In addition, most of these approaches end up producing complex code which is difficult to maintain, evolve and reuse. Such a complexity is the result of tangling the functional behaviour of real-time distributed applications with non-functional aspects. Moreover, some of these solutions usually model resource management for single client-object interactions which does not represent any problem for small applications. However, for large applications the code complexity increases considerably. We have developed both an ADL, called Xelha[Duran00c], and a resource configuration description language (RCDL) [Duran00d] for the configuration of the resources system. The former is concerned with the high-level design of QoS management issues. The latter, in contrast, is a set of aspect languages which provide support for the low-level specification of the system resources. Tool support is offered for the mapping of Xelha specifications to RCDL definitions. Further details on the description languages can be found in the references mentioned above. This paper focuses on the means provided to achieve resource reconfiguration. For this purpose, a resource management framework is introduced which encompasses both a resource model and a task model. The resource model concerns the main
focus of this paper and provides various levels of abstraction for resources, resource factories and resource managers. Therefore, both fine- and coarse-grained resource management are feasible within this model. The latter then models both application and middleware services in terms of task hierarchies. That is, such services are broken into tasks which can in turn be recursively split into sub-tasks. Tasks are then associated with top-level resource abstractions. Such an association determines how both resources and resource management policies are allocated to these services. In addition, we make use of reflection as a means of achieving a separation of concerns of the functional and non-functional behaviour (e.g. resource management). Reflection is a means by which a system is able to inspect and change its internals in a principled way [Maes87]. Basically, a reflective system is able to perform both self-inspection and selfadaptation. To accomplish this, a reflective system has a representation of itself. This representation is causally connected to its domain, i.e. any change in the domain must have an effect in the system, and vice versa. A reflective system is mainly divided into two parts: the base-level and the meta-level. The former deals with the normal aspects of the system whereas the latter regards the system’s representation. The meta-level interface is often referred to as the meta-object protocol (MOP) [Kiczales91]. The paper is structured as follows. Section 2 introduces the overall approach of our middleware platform. Section 3 presents the resource framework. Some extensions for QoS management support are presented in section 4. The implementation of the system’s prototype is then described in section 5. Some experimental results are presented in section 6. Section 7 then provides an overview of some related work. Finally, some concluding remarks are presented in section 8.
2 Middleware Architecture The base-level of the middleware platform concerns the programming structures of the services provided by such a platform. These services are accessed through the base-level interface. In contrast, the meta-level provides the reification of the non-functional properties of this platform, e.g. the resource management of the objects involved in a particular service. The meta-level provides a means to inspect and modify the implementation of the middleware by accessing a well-established meta-interface. For instance, the computational resources involved in a service may be reconfigured by manipulating the operations defined at the metalevel. We adopt the RM-ODP computational model [ISO95] for both the base- and the metalevel. Objects within this model have several interfaces. Not only operational interfaces but also stream and signal interfaces are supported. Moreover, explicit bindings are supported which offer more control over the communication path between object interfaces. Within the OpenORB [Blair01], the meta-space is structured, but not restricted, as a set of four orthogonal meta-models (interface, architecture, interception and resources.) This approach was first introduced by AL-1/D [Okamura92] in order to simplify the meta-interface by maintaining a clear separation of concerns among different system aspects. The interface meta-model allows for the inspection of all interfaces offered by a given component. In addition, the type signature of a component interface may be discovered dynamically. Importantly, interfaces are immutable. That is, it is not allowed to change the internal implementation of an interface. The architecture meta-model deals with the way a composite object is composed, i.e. how its components are inter-connected, and how these connections are manipulated. A component-graph is used for representing composite components, in which components are connected together by edges representing local bindings. The interception meta-model is in charge of introducing additional behaviour to a component interface. This meta-model is realised as dynamic plugable interceptors, which enable the insertion of pre- and post-behaviour. Finally, the resource meta-model is concerned with both the resource awareness and resource management of objects in the platform. A more comprehensive description of the resource meta-model is presented below. Further details of
the overall architecture can be found in the literature. For example, detailed descriptions of the four meta-models can be found in [Blair01].
3 Modeling Resources for Middleware 3.1 Resource Model The most important elements of the resource model are abstract resources, resource factories and resource managers [ReTINA99, Blair99, Duran00a, Duran00b, Durand]. Abstract resources explicitly represent system resources. In addition, there may be various levels of abstraction in which higher-level resources are constructed on top of lower level resources. Resource managers are responsible for managing resources, that is, such managers either map or multiplex higher-level resources on top of lower-level resources. Furthermore, resource schedulers are a specialisation of managers and are in charge of managing processing resources such as threads or virtual processors (or kernel threads). Lastly, the main duty of resource factories is to create abstract resources. For this purpose, higher-level factories make use of lower-level factories to construct higher-level resources. The resource model then consists of three complementary hierarchies corresponding to the main elements of the resource model. Importantly, virtual task machines (VTMs) are top-level resource abstractions and they may encompass several kinds of resources (e.g. CPU, memory and network resources) allocated to a particular task.
team
VTM
thread k thread l
VPi
VTMFact
memory
physical memory
VPj
CPU
(a) A hierarchy of abstract resources
team Fact
memory Fact
VTMSched team Sched
thread Fact
thread Sched
VPFact
VPSched
(b) A factory hierarchy
memory Mgr
(c) A manager hierarchy
Figure 1. A Particular Instantiation of the Resource Meta-model
As an example, a particular instantiation of the framework is shown in figure 1 (note, however, that the framework does not prescribe any restriction in the number of abstraction levels nor the type of resources modelled). At the top-level of the resource hierarchy is placed a VTM, as shown in figure 1 (a), which encompasses both memory buffer and a team abstraction. The team abstraction in turn includes two or more user level threads. Moreover, a user level thread is supported by one or more virtual processors (VPs), i.e. kernel level threads. At the bottom of the hierarchy are located physical resources. In addition, a VTM factory is at the top of the factory hierarchy and uses both a memory and a team factory. The team factory then is supported by both the thread and the virtual processor factory as depicted in figure 1 (b). Finally, the manager hierarchy is shown in figure 1 (c). The team scheduler and the memory manager support the VTM scheduler to suspend a VTM by temporally freeing CPU and memory resources respectively. The thread scheduler in turn allows the team scheduler to suspend its threads. Finally, the VP scheduler supports the preemption of virtual processors. Conversely, this hierarchy also provides support for resuming suspended VTMs. 3.2 The Resources Meta-Object Protocol Most of the operations that constitute the resources MOP are shown in figure 2. This figure also depicts the class hierarchy of the resource framework. Such a class hierarchy may be
used to support a particular instantiation of the resource framework. The Common class is a mixin class for the three class hierarchies (i.e. resources, factories and managers). The interface of this class provides operations to traverse any of the hierarchies by recursively accessing either the lower- or the higher-levels. For instance, the resource hierarchy may be traversed by applying the getLL() operation at the top-level, i.e. the VTM. This operation would be later applied to the lower-level resources, and so on. Access to both the manager and factory of an abstract resource can be obtained through the interface defined by the class AbstractResource. In addition, machines are capable of performing some activity [ReTINA99, Blair99], that is, they receive messages and process them. Thus, machines may be either abstract processing resources (e.g. threads) or physical processing resources (e.g. CPUs). In contrast, Jobs [Blair99] only refer to abstract processing resources since they inherit from the abstract resource class. Both abstract resources and jobs are created by factories as shown in figure 2. In addition, abstract resources are managed by managers. However, since jobs are processing resources, they are managed by schedulers instead. The class hierarchy of factories is rather simple. The AbstResrcFactory class interface exposes an operation for the creation of abstract resources. This operation is also responsible for associating the resource with a resource manager. In case of creating processing resources, a job factory is used and the scheduling parameters should be indicated. This interface also provides the operation getResources() that returns references of the resources created by the factory.
common getLL( ) setLL( ) getHL( ) setHL( ) Machine
Manager AbstractResource
run( )
1..*
maps myManager
resources getManager( ) setManager( ) getFactory( ) setFactory( )
Job
AbstResrcFactory 1..*
creates
resources
myFactory
newResource( ) getResources()
setPolicy( ) admit( ) reserve( ) expel( ) getResources( ) addResource( ) removeResource( )
multiplexes Scheduler
getSchedParam( ) setSchedParam( )
JobFactory creates newResource()
suspend( ) resume( ) schedule( )
Figure 2. Resource Framework Classes
Finally, in respect to the manager hierarchy class, the manager class interface exposes the operation admit() which performs an admission control test that determine whether or not there are enough resources to satisfy a resource request. In a successful case, resources may be reserved by using the operation reserve(). Reservations can then be liberated by invoking the operation expel(). Similar to factories, through the operation getResources(), resource managers are able to retrieve the references of the resources that are mapped or multiplexed. Such references may be included or removed from a manager’s registry by using the operations addResource() and removeResource() respectively. In addition, the operation setPolicy() allows the user to set the management policy of the manager. Moreover, the scheduler allows the user to suspend and resume an abstract processing resource by invoking the suspend() and resume() operations respectively. Lastly, the order of execution of multiplexed resources is determined by the schedule() operation.
3.3 The Task Model The main feature of the task model is that it offers support for the high-level analysis and design of the system’s resource management. More precisely, the task model allows us to define both how system resources are allocated in a distributed system and the resource management policies that are used. We define tasks from two points of view. From the application point of view a task is an activity performed by the system, e.g. transmitting audio over the network or compressing a video image. In contrast, from the programmatic point of view a task is a method invocation sequence which has an amount of resources assigned for its execution. A task may span the boundaries of an object and even those of an address space. The task model is concerned with both application services and middleware services. Thus, we take a task-oriented approach for managing resources in which services are broken into tasks and are accommodated in a task hierarchy. Top-level tasks are directly associated with the services provided by a distributed system. Lower levels of this hierarchy include the partition of such services into smaller activities, i.e. sub-tasks. This approach offers a fine-grained modeling of resource management. Tasks are not necessarily disjoint and may be interconnected. For instance, an object running one task may invoke another object concerned with a different task. Such a method invocation represents a task switching point. In addition, there is a one-to-one mapping between tasks and VTMs within an address space. Hence, a VTM represents a virtual machine in charge of supporting the execution of its associated task. VTMs also represent a higher-level of resource management. They are an encapsulation of the pool of resources allocated for the execution of their associated tasks. Thus, a task switching point corresponds to a change in the underlying resource pool to support the execution of the task that has come into play. VTMs isolate the resources that a service uses to have a better control of the desired level of QoS provided by it. That is, the resources a task uses for execution are localised. Hence, it is straightforward to access the resources of a task when its is under-performing to reconfigure its resources. 3.4 Example of Introspection and Reconfiguration As an example of resource introspection consider the VTM vtmx with similar levels of abstraction to the example presented in section 3.1. This VTM supports the execution of the component codec that is in charge of compressing audio packets in an audio binding. In order to obtain the lower-level resources of this VTM the following operations are performed: team = vtmx.getLL()[‘TEAM’] buffer = vtmx.getLL()[‘BUFFER’] communication = vtmx.getLL()[‘COMMUNICATION’] threads = team.getLL()[‘TEAM’] vpi = threads[i].getLL()
The resources concerning one level of abstraction below the VTM are first obtained. These include a team of threads, memory buffer, and communication resources. Similarly, the getLL() operation is recursively applied to the team and then to a particular thread. As a result, the virtual processor that supports the execution of the team of threads is obtained. Both the scheduler and the factory of a particular thread may be then accessed as follows: threadj = threads[j] threadSchedulerj = threadj.getScheduler() threadFactoryj = threadj.getFactory()
The policy of the thread scheduler is then changed from lottery scheduling to rate-monotonic as an example of resource reconfiguration: threadSchedulerj.setPolicy(‘rate-monotonic’)
It has then been noticed that the virtual processor vpi is under performing. To tackle this situation, the priority of this virtual processor is increased by changing the scheduling parameters as follows: vpi.setSchedParam(new_schedParam)
Later on the application user has solicited an increase in the level of QoS. Thus, another compression scheme is used which notably improves the level of QoS but does require more network resources. A reconfiguration of the network resources is then performed as follows: connections = communication.getLL()[‘CONNECTIONS’] connectionk = connections[k] connectionk.setSchedParam(new_param)
The set of network connections is first obtained. Afterwards, a particular connection is accessed and its QoS parameters are changed. These parameters may include new settings for both the end-to-end delay and the throughput.
4 Extensions for QoS Management In order to provide QoS management support to the resource framework, we further introduce QoS managers, resource servers and resource server proxies, as showed in figure 3. Upon a service request with associated QoS properties, the QoS manager interacts with resources, factories and managers in order to satisfy the required level of QoS. For this purpose, the QoS manager is responsible for looking up the service that best matches the required level of QoS. The QoS manager is also in charge of requesting the appropriate resource managers to perform both the admission control tests and the reservation of resources when the test is successful. QoS managers may be either global or local. There is a single global QoS manager in a system whereas there is one local QoS manager per address space. The latter is only able to attend local requests, in contrast, the former is allowed to carry out requests involving the local capsule and/or remote capsules. Therefore, when a local QoS manager receives a request involving remote parties, the request is forwarded to the global QoS manager. Resource servers are in charge of receiving request from resource server proxies located in remote capsules. The main role of resource servers is to allow remote parties to have access to the local resources, factories, and managers. Hence, these servers delegate the received request to the appropriate entities. Resource server proxies are then basically communication gateways that allow access to remote resources meta-spaces. Therefore, these proxies are able to send requests for the inspection and reconfiguration of remote resources. The MOP of the QoS manager, the resource server and the resource server proxy are depicted in table 1. It should be noticed that the resource server may receive operation requests on resources, factories, managers and the QoS manager. For instance, when the operation getLL(vtm_id) is invoked on the resource server, it localises the actual VTM associated with the identifier and delegates the operation to the VTM. Hence, the lower-level resources of the VTM are obtained. Finally, the identifiers of these resources are retrieved by the resource server and sent back to the Proxy. In addition, a system of management components defined in [Blair00] is used in order to support monitoring and adaptation capabilities in the resource framework. The role of management components basically involves two aspects, namely, monitoring and control [Blair00]. The former includes event collectors, which interface with the underlying implementation, and monitors which detect QoS violations. Interestingly, monitors may be placed anywhere. As a result, not only resource elements may be monitored but also application and middleware components.
Component configuration response (QoS contract) Event collector
Request (service, QoS spec)
Event collector
Application
QoS Manager
Event collector
Resource Server
Resources Monitor Factories Managers
Strategy Selector
Component configuration
Resource Proxy
Event collector Strategy Activator
Middleware General-Purpose OS with RT
Extensions
Figure 3. QoS Management Support for the Resource Framework
Control aspects include then strategy selectors and strategy activators. It is the responsibility of strategy selectors for deciding which strategy to apply upon the occurrence of a QoS violation. Strategy activators are then in charge of realising the adaptation strategy by providing the detailed implementation of this strategy. Hence, strategy activators are able to execute actions that change the value of the parameters of an operation performed by a component. Strategy activators have access to resources, factories and managers in order to perform resource reconfiguration. These activators may also have access to remote resources meta-spaces through resource server proxies. Moreover, reconfiguration of both application and middleware components configurations may be carried out by strategy activators. Finally, the assumption for the general-purpose operating system is that the ability for dispatching threads in a pre-emptive fashion is supported. MOP of the QoS Manager, Resource Server and Resource Server Proxy Operation Signature Description Operation on QoS Managers getService(serviceType, Look up the service with the requested level of QoS, perform parameters, qos_spec) admission control test and reserve resources Operations on Resource Servers startServing() Get the resource server ready for receiving requests stopServing() Stop the resource server from receiving requests operation([entity_id,] Apply the operation with the given parameters on the indicated operation_param) entity Operations on Resource Server Proxies connect(capsule) Connect the proxy to the given capsule close() Close the current connection operation([entity_id,] Forwards the request of the execution of an operation to the operation_param) resource server
Table 1. The MOP of the QoS Manager, Resource Server and Resource Server Proxy
Importantly, in order to assure that no race conditions take place when the admission control is performed and resources are reserved or reconfigured, both the strategy activator and the
QoS managers must acquire the resources meta-space lock before carrying out any action. There is a resources lock per address-space. Hence, local QoS managers only need to acquire the local lock. The global QoS manager and the strategy activator, however, must acquire the lock of all the involved parties. Furthermore, in order to avoid potential deadlock problems, only a single strategy activator can be defined in the system. In addition, the activator must be located in the same capsule as the global QoS manager. Thus, the first in acquiring the local lock is entitled to try to obtain any required remote locks.
5. Implementation 5.1 Overall Approach A prototype of the resource framework has been implemented in Python [Watters96], which is an ideal language for rapid prototyping. Similar to Java, Python modules are translated into byte-code to speed up the interpretation process. Moreover, since Python can easily be extended with C programs, Python’s applications can dramatically improve their performance by programming the most critical modules in C. In addition, the resource framework’s prototype was integrated into OOPP [Andersen00], as shown in figure 4, which is a Python implementation of the OpenORB framework. Hence, the OOPP has been extended with resource management capabilities. The extended version of OOPP is then able to perform admission control test upon resource requests. Importantly, the OOPP has been provided with support for both inspection and reconfiguration of resources. In addition, the OOPP prototype was extended to provide a point-to-point audio communication service. The service can be requested with a particular QoS specification. Resources are reserved according to the QoS level requirements. Adaptation mechanisms are also supported in case unexpected changes are introduced into the environment. These mechanisms involve the introduction of different compression strategies and the repartition of CPU in the system. In addition, the Python interpreter was then layered on top of GOPI [Coulson98] which is a micro-ORB implemented in C that offers support for distributed multimedia applications. In particular, we make use of the thread and memory management features offered by GOPI. A two-level scheduling model was implemented whereby the top-level is in charge of allocating a portion of the CPU to VTMs. Finally, the system was deployed on a cluster of ULTRA 5 Sun SPARC stations running Solaris 2.5.8. OpenORB Resource Framework Python Interpreter
GOPI Sun SPARC/Solaris 2.5.8
Figure 4. Implementation of the Resource Framework
5.2 Integrating the Resource Framework into OOPP We have integrated a particular instantiation of the framework similar to that described in section 3.1 into OOPP. VTMs are top-level processing resources that include both abstract resources (i.e. buffer memory) and abstract processing resources (i.e. threads). A distributed VTM encompasses one local VTM and two or more remote VTMs. Remote VTMs are abstractions that keep information of the VTM residing in a remote site in terms of both the id of such a VTM and a proxy of the resource server that resides in the remote capsule. The interface of the resource server exposes operations that delegate the actions of inspection and reconfiguration of resources to the appropriate entities (e.g. VTMs, managers, etc). Therefore,
any of the three hierarchies of the resource meta-space may be remotely accessed by the proxy. Importantly, task switching points are implemented as interceptors by using the interception meta-model. Hence, invocation calls are intercepted whereby a task switch is performed in case the current task is different from the task associated with the invoked method. The request is then attended by the resource pool associated with the new task. In addition, objects within this ORB may have several interfaces which can be bound to other matching interfaces whereby the exported methods correspond to the imported methods of the adjacent interface. Remote bindings permit connecting interfaces residing in different capsules. Three styles of binding are provided. Operational bindings are adequate for remote method invocation whereas signal bindings are suitable for one-way notification. Lastly, stream bindings provide support for stream communication. In addition, capsules represent an address space and allows objects to invoke other objects residing in a different capsule as if they were in the same capsule. The name server allows the user to register both interfaces and capsules and, as a consequence, to access them remotely. Finally, the node manager allows several capsules to coexist within the same node. It is worth mentioning that we are aiming for an open middleware framework and interoperability with current standards is not a primary concern for this research. 5.3 The Two-Level Scheduling Model A two-level scheduling model for Python was implemented on top of GOPI’s thread library. This scheduling model was implemented as part of the resource model. More precisely, the scheduling model is employed for the scheduling of VTMs. As mentioned in section 3.1, the VTM scheduler is responsible for scheduling the processing resources of the VTMs. For this purpose, the VTM scheduler schedules the thread schedulers associated with such VTMs. The VTM scheduler uses a priority scheduling structure similar to [Nahrstedt98] and extends it to a two-level scheduling structure. The two scheduling model operates within an address space, however, this approach can also be applied at the operating system level, as shown in [Nahrstedt98], to ensure the availability of a portion of the CPU in a multi-shared environment. The VTM scheduler runs at the highest possible fixed-priority as shown in figure 5. Thread schedulers are then located at the second level of scheduling with the second highest priority. In addition, active threads are placed at the third highest priority whereas a sentinel thread remains on a level below. This thread is required by GOPI to make sure that there is always a thread to run. Finally, non-active threads are placed in a waiting list a level below the sentinel. The VTM scheduler wakes up periodically, every 5 ms, to dispatch the next thread scheduler by un-blocking its associated thread and then goes to sleep. The thread scheduler then removes the threads located at the third level and introduces them into the waiting list at the bottom level. Following this, the thread scheduler selects one or more threads from the waiting list and places them at the third level. Afterwards, the thread scheduler blocks itself and the active threads start executing. In our current implementation, the first level of scheduling supports both rate-monotonic [Liu73] and lottery scheduling [Waldspurger94] whereas the second scheduling level uses a round-robin policy. Scheduling Structure Thread Priority 0 VTM Scheduler Thread Scheduler 1 2 Active Threads 3 Sentinel 4 Waiting Threads Figure 5. Priority Scheduling Structure.
The rate-monotonic policy is implemented by generating a dispatch table according to both the period and the execution time of a VTM. The dispatch table, which is implemented as a Python list, defines the order of execution of VTMs. The lottery scheduling policy was implemented by assigning notional lottery tickets to VTMs according to their parameter of CPU percentage usage. For instance, a VTM with a portion of 20% of CPU may be assigned 20 nominal tickets numbered from 1 to 20. Another VTM with a CPU portion of 40 % would have tickets numbered from 21 to 60; and so on until a total amount of 100 tickets are assigned. A ‘lottery’ is hold at each scheduling point by generating a random number between 1 and 100. The VTM holding the winner ticket then obtains the right to execute until the next scheduling point. Lastly, the round-robin policy is implemented by the thread library of GOPI and is not further described.
6. Experiments and Results We have carried out a number of experiments in order to test the resource framework’s prototype. The experiments were carried out on two networked Ultra 5 SUN SPARC stations running Solaris 2.5.8. The experiments consider a system running three audio connections between capsule 1 and capsule 2 as shown in figure 6. Two audio connections are sharing the task transmitAu_V1 whereas only one connection is participating in task transmitAu_V2. The internal configuration of each connection includes an audio compressor and a stub in the source side whereas the counterpart involves a stub and a decompressor as shown in figure 7. Both stubs are connected by a stream connector. Three compression schemes are supported by both the compressor and the decompressor, namely GSM, MELP and LPC10 [Spanias94, Spanias00, Spanias01]. In addition, the bindings include a QoS management structure which involves both an automaton component and an activator component. The former is in charge of monitoring packet losses. On the occurrence of a QoS violation, it is also responsible for deciding which strategy to apply. Such strategy is then realised by the activator component. Task path Task starting point
Task end point transmitAu_V1 Audio Binding 1 source1
sink1
transmitAu_V1 Audio Binding 2 source2
sink2
transmitAu_V2 Audio Binding 3
Capsule 1
source3
sink3
Capsule 2
Figure 6. Scenario 1
The initial resource configuration is specified by RCDL descriptions (not shown for the sake of brevity) which define a pool of threads and the amount of memory buffer for tasks transmitAu_V1 and transmitAu_V2. The RCDL definitions of capsule 1 determines an execution of 10 ms every 120 ms for both tasks transmitAU_V1.send and transmitAU_V2.send. In addition, the task default has allocated 10 ms in a period of 20 ms.
Task path Task starting point
Task end point
transmitAu
compressor
srcStub
sinkStub
decompressor sink
source transmitAu.send
streamConn
activatorConn
COLLECT
activator
transmitAu.receive
automaton
Figure 7. Structure of a Binding Component
In contrast, the RCDL definitions of capsule 2 prescribe the same portion of CPU for all tasks which is equivalent to 10 ms out of 40 ms. The VTM scheduler uses the rate-monotonic policy by default. The unit of execution provided by the scheduler is 5 ms. The average execution times of the instances of task transmitAu_V1 are then indicated in RCDL descriptions (not shown for the sake of brevity) which define the average execution times for the different compression algorithms. A scenario where overloaded network conditions are detected is simulated by introducing a mechanism that randomly drops packets. In case the packet loss is greater than 40 % the automaton raises a QoS violation and selects an adaptation strategy, which is realised by the activator. Upon the detection of this condition, the activator changes the GSM compression algorithm that has a throughput requirement of 13 Kbit/sec to MELP, which provides a higher compression rate that only requires 2.4 Kbit/sec. This is only feasible, however, if there is spare processing capacity as the computation time requirement by a MELP codec is 3.5 ms for the encoder and 2 ms for the decoder. The automaton inspects the resource meta-model to collect statistics on the available processing resources. However, the admission test fails in capsule 1 as the remaining CPU capacity is not enough to cover the demands of the new compression scheme. The task TransmitAudio_V2, however, has some spare CPU resources and its criticality is lower than that of task TransmitAudio_V1. It is worth mentioning that the current implementation makes an implicit reservation of 1.5 ms for the overhead imposed by the second scheduling level. Therefore, a CPU reconfiguration is performed as follows: proxy.cpu_reconfig(’transmitAu_V1.send’,’transmitAu_V2.send’,120.0,120.0,15,5)
whereby the task transmitAu_V1.send is assigned 15 ms of CPU every 120 ms whereas the execution time for task transmitAu_V2.send is set down to 5 ms in a 120 ms period. In addition, the GSM strategy is disabled while the MELP encoder and decoder are activated for each of the three audio binding. The graphs in figure 8 show the measurements of inter-packet time taken on the output of the decoders of the two connections part of task TransmitAudio_V1 when the compression algorithm is changed to the MELP scheme and no CPU reconfiguration is performed.
500
500 450
450
400
400
350
350
300
300
250
250
200
200
150
150
100
100
50
50
0
0
Packet Number
(a) Audio Binding 1
P ack et N u mber
(b) Audio Binding 2
Figure 8. Inter-Packet Time Measurement for the Audio Binding 1 and the Audio Binding 2 in Scenario 1 without CPU Reconfiguration
The processing resources of task transmitAu_V1.send become clearly overloaded since the CPU requirements for the MELP encoder cannot be fulfilled by this task. As a result of the activation of the codecs, noticeable jitter over 120 ms (equivalent to 6 GSM packets) is introduced. For a given packet, jitter is computed as follows: jitter = inter-packet time – period
where period is the transmission rate i.e. 120 ms. Note that the receiver, in both tasks, has a higher rate in order to avoid packet losses. 500 450 400 350 300 250 200 150 100 50 0
Packet Number
Figure 9. Inter-Packet Time Measurement for the Audio Binding 3 in Scenario 1 without CPU Reconfiguration
In contrast, the audio binding 3 belonging to task transmitAu_V2 remains unaffected by the new processing demands required by the MELP codecs of both the audio binding 1 and the audio binding 2, as shown in figure 9. The reason of this is that, in spite of the activation of the MELP scheme for binding 3, the task transmitAu_V2 still has enough CPU resources to cover the new demands. Note, however, that some jitter is introduced for a short period of time. Such a period corresponds to the time when packets are dropped. On the other hand, when CPU resources are dynamically reconfigured, the amount of jitter is considerable reduced in both the audio binding 1 and the audio binding 2, as shown in figure 10. The packets with a time-interval higher than 300 ms and less than 700 ms denote the lapse when packet losses are induced. As soon as this condition is detected the resource reconfiguration takes place and the time-interval of packet arrivals come back to normal.
1000 1000
900
900
800
800
700
700
600
600
500
500
400
400
300
300
200
200
100
100
0
0
Packet Number Packet Number
(a) Audio Binding 1
(b) Audio Binding 2
Figure 10. Inter-Packet Time Measurement for the Audio Binding 1 and the Audio Binding 2 in Scenario 1 with CPU Reconfiguration
There are a few packets with a jitter around 40 ms that can still be seen in both graphs. Such spurious delays are due to the fact that the sender and the receiver are not synchronised. Thus, these packets represent the case where the receiver thread is unscheduled just before a packet arrives. In addition, both graphs depict the occurrence of a packet with an inter-packet time higher than 950 ms before packet losses are induced. The reason of this is that our current implementation has not control on the operating system scheduler. As a result, some processes might unexpectedly run such as those background process periodically executed by root.
7. Related Work There has been some interest in the development of resource models for distributed object systems. The QuO project [Pal00] provides a framework for the specification of QoS of CORBA object interactions. QoS aspects are specified in QDL which specify QoS aspects in various description languages. A code generator is used to produce the code of contract objects, delegate objects and system objects. The former represent operating regions and application service requirements. Delegate objects support adaptive behaviour upon method call and return. System condition objects interface to resources, mechanisms and ORBs in the system. However this work focuses on client-object interactions whereas our work additionally covers resource management of coarser-grained interactions. The ERDoS project [Chatterjee99] offers a generic and comprehensive resource framework. It provides an extended CORBA IDL that partially captures the resource framework. Similar to our work, resource management is performed at the middleware level. A distributed system is described within three base-level models whereas our approach is based on a multi-model reflection framework. The Global Resource Management System (GRMS) [Huang98] provides middleware mechanisms for QoS negotiation and adaptation. The GRSM offers a generic resource model where heterogeneous resource management components are modeled uniformly. A two-phase negotiation and adaptation protocol is used to perform a control admission test across multiple nodes and propagating either a commit or an abort command. In addition, the protocol supports the shrinking of the executing sessions’ QoS, in case no enough resources are available to attend the demands of all sessions. Expansion of the executing sessions’ QoS, applied at the end of the negotiation phase, is also supported in order to maximise the application QoS. However, the framework only targets applications that explicitly express their desire of changing the QoS mode of operation. That is, no support is provided for applications that require both to monitor the dynamic changes introduced to their environment and react accordingly. TAO [Schmidt97] is a high-performance real-time ORB. TAO is CORBA-compliant and has introduced several extensions to this standard in
order to overcome the shortcomings of conventional ORBs regarding efficiency issues. Nevertheless, little support is provided for dynamic resource reconfiguration. The Realize resource management system [Kalogeraki00] offers a framework that supports dynamic scheduling. Notably, the framework provides tasks as schedulable entities, which may span processor boundaries. More specifically, tasks are modelled as sequences of method invocations whereby request messages carry scheduling parameters along the involved nodes. The idea of tasks in this framework is similar to distributable threads, which are introduced by the OMG’s Dynamic Scheduling proposal (see below). ObjectTime [Lyons98] is a tool that implements UML for real-time constructs [Selic98]. Specifically, the modelling structure of this tool comprises three elements, namely, capsules, ports and connectors. The former are actors (i.e. active objects) that communicate with each other through ports. Such ports are objects that implement a particular interface. Finally, connectors represent signal-based communication channels. However, no means is provided to model resources. A generic framework for modelling resources with UML [Selic00] has been developed and a proposal [OMG00] has been submitted for the OMG adoption. Both, physical and logical resources at various levels of abstractions may be represented within this model. QoS management is modelled in terms of QoS contracts, resource managers and resource brokers. Other efforts in this area include that performed by the OMG which has recently adopted a real-time specification for CORBA [OMG99a]. This specification presents a platform independent priority scheme and defines a propagated priority model in which CORBA priorities are propagated across address spaces. In contrast with our work, this adopted standard does not support dynamic scheduling. The OMG’s Dynamic Scheduling proposal [OMG00] has been issued to overcome the limitations imposed by RT-CORBA in terms of dynamic scheduling. Interestingly, a distributable thread is defined as a schedulable entity that may transverse object boundaries and even node boundaries. This entity replaces the concept of activity introduced in RT-CORBA. A distributable thread is basically characterised by a unique system-wide id and a set of scheduling parameters. An application may encompass many distributable threads, which execute concurrently and asynchronously by default. Importantly, a distributable thread carries the scheduling parameters set by the application along the visited nodes. However, the proposal does not support various levels of abstractions for modelling resources. In addition, the concept of task in our framework is more comprehensive since it can be related to various types of resources whereas the proposal constraints the use of distributable threads to CPU resources. Finally, the approach followed by the XMI and the MOF [OMG99b] focus on the interchange and management of the meta-information of a system respectively. Complementary to the work presented here, the latter standard has been used in [Costa00a] to define a type system for a repository of middleware configurations.
8. Concluding Remarks We have presented a resource management framework within the context of the OpenORB architecture. The resource framework encompasses a resource model and a task model. The former allows for both resource introspection and resource reconfiguration. In addition, system resources are explicitly represented at various levels of abstraction. The first-class treatment of resources and resource managers provides a more natural way and a higher-level of reasoning about resource management. The task model then allows for the high-level analysis and design of the system’s resource management. Notably, resource management of both coarse- and fine-grained interactions is feasible within the model. The former is achieved by managing top-level tasks which may transverse more than one address space. The latter is obtained as a result of employing task partitioning. Both the task model and the resource model are tightly coupled and allow for a high-level of resource management. The task model is concerned with the logical flow of component interactions which are mapped to resource
pools (e.g. VTMs). The resource model is then in charge of providing VTMs abstractions for the representation of the physical resources. In addition, the use of reflection permits us to separate functional aspects from non-functional ones, thus, leading to clearer and more reusable code. Our experimental results have shown the feasibility of the approach. Although Python is not an efficient language, the Python prototype has proved to be efficient enough for small distributed multimedia applications. A more efficient implementation of the resource framework could be used for larger applications. The work has mainly been focused on distributed multimedia systems. However, our approach is not constrained to this area and can be extended for other application domains. For instance, storage resources may be modelled for both database systems and persistence services. In contrast, other higher-level middleware services, such as the transaction service, represent a more complex issue and would require further study. Future work concerns addressing the issue of broadening the number of computational resources modelled along with the incorporation of other resource management policies. At present, support for CPU reservations is only provided at the address-space level. Further work involves extending this support to the operating system level. Finally, consistency check mechanisms should also be developed to ensure a reconfiguration does not bring the system to an inconsistent state.
Acknowledgements Hector A. Duran-Limon would like to thank his sponsor, the Autonomous National University of Mexico (UNAM). Particular thanks are also due to Jean-Bernard Stefani and his group at CNET.
References [Andersen00] Andersen A., Gordon S. Blair, Frank Eliassen. “A Reflective Component-Based Middleware with Quality of Service Management”. In Proceedings of the Conference on Protocols for Multimedia Systems (PROMS2000), Cracow, Poland, October, 2000. [Blair00] Blair, G.S. et al., “Supporting Dynamic QoS Management Functions in a Reflective Middleware Platform”, IEE Proceedings Software, vol 147, no.1, Feb. 2000, pp. 13-21. [Blair99] Blair, G.S., Costa, F., Coulson, G., Delpiano, F., Duran-Limon, H., Dumant, B., Horn, F., Parlavantzas, N., and Stefani, J-B. “The Design of a Resource-Aware Reflective Middleware Architecture”, In Second International Conference on Reflection and Meta-level architectures (Reflection’99), St. Malo, France, July 1999. [Chatterjee99] S. Chatterjee, B. Sabata, and M. Brown, "Adaptive QoS Support for Distributed, Java-based Applications". In the Proceedings of the IEEE International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC), St-Malo, France, May 1999. [Costa00a] Costa, F., Blair, G.S., “The Role of Meta-Information Management in Reflective Middleware”, Proceedings of the ECOOP’98 Workshop on Reflective Object-Oriented Programming and Systems, ECOOP’00 Workshop Reader, Springer-Verlag, 2000. [Coulson98] Coulson, G., "A Distributed Object Platform Infrastructure for Multimedia Applications", Computer Communications, Vol. 21, No. 9, pp 802-818, July 1998. [Duran00a] Duran-Limon H. and Blair G. A Resource Management Framework for Adaptive Middleware. In 3th IEEE International Symposium on Object-oriented Real-time Distributed Computing (ISORC’2k), Newport Beach, California, USA, March 2000. [Duran00b] Duran-Limon H. and Blair G. Configuring and Reconfiguring Resources in Middleware. In the 1st International Symposium on Advanced Distributed Systems (ISADS’2000), Gdl, Jalisco, Mexico, March, 2000. [Duran00c] Duran-Limon H. and Blair G. Specifying Real-time Behaviour in Distributed Software Architectures. In the 3th Australasian Workshop on Software and System Architectures, Sydney, Australia, November, 2000. [Duran00d] H. Duran-Limon and Gordon Blair. The Importance of Resource Management in Engineering Distributed Objects. In the 2th international workshop on Engineering Distributed Objects (EDO2000), California, USA, November, 2000. [Huang98] J. Huang, Y. Wang, F. Cao, "On developing distributed middleware services for QoS- and criticalitybased resource negotiation and adaptation", In the Special Issue on Operating Systems and Services, Journal of Real-Time Systems, 1998. [ISO95] ISO/ITU-T. Reference Model for Open Dist. Processing. International Standard ISO/IEC 10746, 1995. [Kalogeraki00] V. Kalogeraki, P. M. Melliar-Smith and L. E. Moser. Dynamic Scheduling of Distributed Method Invocations, In 21st IEEE Real-Time Systems Symposium , Orlando, Florida (November 2000).
[Kiczales91] Kiczales, G., des Rivières, J., and Bobrow, D.G., “The Art of the Metaobject Protocol”, MIT Press, 1991. [Liu73] C. Liu and J. W. Layland. Scheduling Algorithms for Multiprogramming in a Hard Real Time Environment. JACM, 20(1): 46-61, 1973. [Lyons98] Lyons A., “Developing and debugging Real-Time Software with ObjecTime Developer. Available at http://www.objectime.com [Maes87] Maes, P., “Concepts and Experiments in Computational Reflection”, In Proceedings of OOPSLA’87, Vol 22 of ACM SIGPLAN Notices, pp 147-155, ACM Press, 1987. [Nahrstedt98] Klara Nahrstedt. Hao hua Chu, and Srinivas Narayan. QoS-aware Resource Management for Distributed Multimedia Applications. Journal of High-Speed Networking, Special Issue on Multimedia Networking, 7:227-255, 1998. [Okamura92] Okamura, H., Ishikawa, Y., Tokoro, M., “AL-1/d: A Distributed Programming System with MultiModel Reflection Framework”, Proceedings of the Workshop on New Models for Software Architecture, November 1992. [OMG00] Joint Initial Proposal for a UML Profile for Schedulability, Performance, and Time. http://www.omg.org/, 1999, Object Management Group. [OMG99a] Real-Time CORBA 1.0 Specification, http://www.omg.org/, 1999, Object Management Group. [OMG99b] Meta Object Facility Specification. http://www.omg.org/, 1999, Object Management Group. [OMG00] Dynamic Scheduling Revised Submission, http://www.omg.org/, 2000, Object Management Group. [Pal00] P. Pal, J. Loyall, R. Schantz, J. Zinky, R Shapiro and J. Megquier. Using QDL to Specify QoS Aware Distributed (QuO) Application Configuration. In 3th IEEE International Symposium on Object-oriented Realtime Distributed Computing (ISORC’2k), Newport Beach, California, USA, March 2000. [ReTINA99] ReTINA, “Extended DPE Resource Control Framework Specifications”, ReTINA Deliverable AC048/D1.01xtn, ACTS Project AC048, January 1999. [Schmidt97] Schmidt, D., D. Levine, and S. Mungee, “The Design of the TAO Real-Time Object Request Broker”, Computer Communications Journal, Summer 1997. [Selic98] Selic B., Rumbaugh J., “Using UML for Modeling Complex Real-Time Systems”, available at http://www.objectime.com [Selic00] Selic B., “A Generic Framework for Modeling Resources with UML”, IEEE Computer, Special Issue on Object-Oriented Real-time Distributed Computing, edited by Eltefaat Shokri and Philip Sheu, June 2000. [Spanias00] A. S. Spanias, "Speech coding standards," Chapter 3, pp. 25-44, Academic Press, Ed: G.Gibson, ISBN 2000 0-12-282160-2 [Spanias01] Andreas Spanias," Speech Coding for Mobile and Multimedia Applications,"Short Course Notes, April 2001 [Spanias94] A. S. Spanias, "Speech coding: a tutorial review," Proc. of the IEEE, vol. 82, no. 10, pp. 1441-1582 (42 pages), October 1994. [Waldspurger94] Waldspurger, C.A. and Weihl, W.E., “Lottery Scheduling: Flexible Proportional-Share Resource Management”, Proc. First USENIX Symposium on Operating Systems Design and Implementation, November, 1994. [Watters96] Watters, A., van Rossum, G., Ahlstrom, J., "Internet Programming with Python", Henry Holt (MIS/M&T Books), September 1996.
