Quality of Service Negotiation for Distributed, Dynamic Real-time ...

Quality of Service Negotiation for Distributed, Dynamic Real-time Systems 1

2

1

2

Charles D. Cavanaugh , Lonnie R. Welch , Behrooz A. Shirazi , Eui-nam Huh , and 1 Shafqat Anwar 1

Computer Science and Engineering Dept. The University of Texas at Arlington Box 19015, Arlington, TX 76019-0015 {cavan|shirazi|anwar}@cse.uta.edu 2 School of Electrical Engineering and Computer Science Ohio University Athens, OH 45701-2979 {welch|ehuh}@ace.cs.ohiou.edu

Abstract. Dynamic, distributed, real-time systems control an environment that varies widely without any time-invariant statistical or deterministic characteristic, are spread across multiple loosely-coupled computers, and must control the environment in a timely manner. In order to ensure that such a system meets its timeliness guarantees, there must be a means to monitor and maintain the quality of service in the system. The QoS manager is a monitoring and diagnosis system for real-time paths, collections of time-constrained and precedence-constrained applications. These applications may be distributed across multiple, heterogeneous computers and networks. This paper addresses the QoS negotiation features of the QoS manager and its interaction with the middleware resource manager. The major contributions of the paper are the negotiation algorithms and protocol that minimize the impact on the other paths’ QoS while maximizing the unhealthy path’s QoS. The approach and algorithms for QoS negotiation are presented.

1 Introduction Dynamic, distributed, real-time systems possess three characteristics. First, the environment that they control is not deterministic and cannot be characterized by time-invariant statistical distributions. Second, the system is spread across multiple loosely coupled computers. Third, the system must control the environment in a timely manner. Existing solutions for monitoring real-time systems [1] and for realtime scheduling are usually based on the assumption that the processes have worstcase execution times. In dynamic environments, such as air traffic control [2], robotics, and automotive safety, this assumption does not hold [3]. The dynamic real-time path [4][5] (Fig. 1) is a collection of time-constrained and precedence-constrained applications. These applications may be distributed across multiple, heterogeneous computers and networks. The QoS manager’s tasks are to monitor path health, diagnose the causes of poor health, and request computation and communication resources to maintain and restore health.

Compute subpaths

Path 3 Guide

Communication subpaths

Path 2 Initiate

Path 1

operator

Assess

sensors

filter/sense

evaluate & decide

act

actuators

Fig. 1. Path composition

The problem of mapping applications to resources is to assign resources to consumers such that the delivered QoS meets or exceeds the QoS requirement (if possible). If this is not possible, some of the resources that are in use by a low criticality real-time application may need to be diverted to a high criticality real-time application. The QoS manager and resource manager must negotiate a solution that is mutually acceptable. QoS negotiation is the process of the QoS manager and the resource manager trading off resources for some applications while improving the QoS of the applications having higher criticality. The rest of this paper is organized as follows: the QoS negotiation architecture and approach are explained in Section 2, the negotiation algorithms and protocol are presented in Section 3, a sample experiment using manual techniques to illustrate QoS negotiation is shown in Section 4, related work is summarized in Section 5, and a summary and statement of future work is in Section 6.

2 QoS Negotiation Architecture and Approach The QoS negotiation architecture is presented in Fig. 2. The QoS monitor’s job is to combine the monitored data into QoS metrics for the path and applications and to translate and pass along relevant application load and resource usage information. The analyzer’s function is to detect QoS violations and calculate trends for QoS metrics, load, and resource usage. The diagnosis component determines the causes of the QoS violations by recognizing conditions that indicate a particular malfunction. The negotiator has two functions. First, it selects actions that will remedy the malfunctions and requests resources for applications if necessary. Second, it negotiates the highest possible QoS with the resource manager when the resource manager indicates that resource availability does not allow a certain action or resource request to be carried out. Negotiation involves trading off some actions for alternative actions that provide the highest possible QoS assurance under the resource availability constraints. The resource manager obtains current utilization levels for communication and computation resources from host monitors. Moreover, resource unification is required to map heterogeneous resource requirements into available target host. Then, the RM finds resources that meet (unified) resource requirements. If the hosts are feasible, then it predicts queuing delays to analyze schedulability.

QoS prediction will result from candidate reallocation actions. Finally, resource allocation selects and performs a reallocation action (through program control and startup daemons) based on predicted QoS. A new selection technique is used to guarantee the customer’s QoS.

Diagnosis Violations and QoS/load trends

Negotiator

Causes of violations

Resource requests and responses

Analyzer QoS Monitor

QoS and resource usage

Timing and resource usage

Reallocation actions

Operating system process control

Applications

Resource Manager Program Control

Process-control messages

Startup Daemon

Fig. 2. QoS Negotiation Architecture

The three phases of QoS negotiation correspond to the three phases of diagnosing poor path health: path-local, resource-local, and global diagnosis. During phase I, path-local diagnosis, the QoS manager requests for allocation actions involving the unhealthy subpaths that it identifies. For example, one application within a path may be unhealthy; and the QoS manager would request that it be scaled up. During phase II, resource-local diagnosis, the QoS manager requests actions involving any software that is sharing resources with the unhealthy path. For example, the QoS manager may request that some competing application program be moved off a host. The QoS manager does not need to know the specific application program that is involved, as it is the resource manager’s responsibility to maintain the system resources. During phase III, global diagnosis, the QoS manager requests actions that involve any resource. For example, the QoS manager may request that a less critical application be moved in order to free up space on a host that is not currently in use by the unhealthy subpath. The resource manager is responsible for finding the best host for the application or path while balancing the load among other paths and applications. The three phases of negotiation are illustrated in the following scenario: QM: Application x on host A is unhealthy and using 20% of CPU. Phase I: can you migrate it to another host? (QM adds action to list of attempted actions.) RM: No. No combination of host idle times adds up to 20%. Provide QoS information, ranked application actions, and resource usage.

QM: (Marks previous action as unsuccessful.) Phase II: can you move competing application y (also on host A), which uses 15% of CPU to another host (to free up 15% of CPU on host A)? (QM adds action to list of attempted actions.) RM: No. No combination of host idle times adds up to 15%. Phase III: I can free up resources on host A by moving a less-critical application to a host with the lowest utility. (RM carries out action.) QM: (Marks previous action as successful.)

3 QoS Negotiation Algorithm and Protocol The QoS manager and resource manager maintain high QoS and manage resources, respectively. Whenever there is a conflict between obtaining enough resources to ensure high QoS and providing enough resources to the rest of the software, the QoS manager and resource manager negotiate a solution. To do this, both need algorithms to work toward their goals as well as a protocol for them to communicate with each other. The flowcharts of the algorithms illustrate the algorithms, and the communication steps show the protocol. The following are the steps that the QoS manager takes once it detects a QoS violation. A flowchart of the process is shown in Fig. 3. First, the QoS manager identifies unhealthy computation and communication subpaths. Depending on the phase of negotiation (path-local, resource-local, or global), and the constraints on allowable actions, the QoS manager then selects actions to remedy the unhealthy subpaths. Each subpath has a resource requirement that is proportional to the slowdown that the subpath is experiencing. The slowdown is the ratio of the current subpath latency to the subpath’s minimum latency for the same data stream size while on the same resource. For example, if the current subpath latency is 0.4 seconds at a data stream size of 1,000 on a particular resource, and the lowest latency that it has experienced in that same situation is 0.3 seconds, then the subpath’s slowdown is 0.4 / 0.3, or approximately 1.333. This implies that it requires (133% - 100%) or 33% more resources to run at its best. The slowdown is due to contention, so moving it to another resource is a likely solution. The QoS manager ranks the actions based on their resource requirements in descending order and groups actions that involve moving subpaths off a particular host and actions that involve replicating a particular subpath (if it is replicable). The groups are automatically ranked, since the groups are made from the sorted list. Once action selection is complete, the QoS manager requests resources by sending to the resource manager the ranked action requests (one from each group) along with the criticalities, current latencies, and resource usage information. If the resource manager responds that it can carry out the action, then the QoS manager monitors the stability of the system once the actions are carried out to ensure that QoS is indeed improved. However, if the RM cannot do the action, then the RM sends a negotiation request to the QM. The QoS manager responds by sending out the next ranked action in each group, or it goes to the next phase of negotiation. The RM responds with a level of degradation in the QoS that is to be expected by the QM. The QoS manager calculates the slowdown that would be associated with the degradation and derives a

benefit value for the path from it. If the benefit is at least as favorable as the QM requires, then the QoS manager responds to the RM with an acknowledgement; otherwise, the QoS manager proceeds to the next phase of negotiation.

Fig. 3. QoS manager QoS negotiation algorithm

The steps that the resource manager takes to negotiate with the QoS manager and to allocate resources are listed below. A flowchart of the process is shown in Fig. 4. 1. Find a feasible host corresponding to resource needs 2. If a host is feasible, then do step 4 3. Else do step 8 4. Predict queuing delay and execution time on feasible hosts 5. If the task with predicted response time is schedulable, then do step 7 6. Else do step 8 7. Predict QoS and allocate it best host and exit 8. Send “QoS negotiation requests” to all QoS managers 9. Receive path QoS information and ranked list of actions and applications’ resource usage from each QM 10.Calculate current utility value of each path 11.Select negotiable paths based on the minimum utility value 12.Calculate host utility values and find the host, Hj, with the minimum utility

13.Select application, ai, in the ranked list of recommended actions 14.Test feasibility of allocating the application, ai, on the host, Hj. 15.If not feasible, then pick next path and do step 11 16.If feasible, then recalculate utility value of path 17.If utility value of each path is less than threshold of utility value of each path then do step 4 18.Else allocate the violated application to the host that has the minimum utility value

Fig. 4. Resource Manager QoS Negotiation Algorithm

4 Experimental Results Sample experimental results were obtained by specifying two DynBench [6] periodic paths in the spec language: a higher criticality sensing path, D:H:Higher_Sensing, and a lower criticality sensing path, D:L:Lower_Sensing. These paths were started simultaneously, and the experiment generator was used to bring the data stream size (the load) to 1600 tracks for each path. Then, the filter and ED applications of each

path were manually replicated to simulate the QoS manager’s requesting that they be scaled up. The latency was brought down at that point. However, 500 more tracks were added to each path in order to overload the paths again. When no action was taken, the system became unstable, despite the fact that the loaded applications in both paths were already replicated. All four available hosts were in use. This instability is evident on the left-hand side of Fig. 5. Negotiation was simulated by manually moving the higher criticality path’s filter and ED replicas to a more powerful host, named texas. In addition, resources were taken away from the lower criticality path by terminating the additional replicas of the lower criticality path’s filter and ED applications, resulting in a normal QoS for the higher-criticality path (C) and a degraded QoS for the lower-criticality path (D), as shown on the right-hand side of Fig. 5. This combination of manual actions simulates the behavior of QoS negotiation and thus serves as a prototypical experiment. The scenario is a case by which an implementation of the negotiation algorithm should be tested.

Fig. 5. Instability caused by overload (2100 tracks per path), without negotiation (left). The higher (A) and lower (B) criticality paths are fluctuating. Stability restored after negotiation (right). The higher criticality path experiences normal QoS (C); the lower criticality path experiences degraded QoS (D)

5 Previous Work in QoS Negotiation To summarize the related work, the related work in negotiation is narrowly defined. The DeSiDeRaTa project promotes a broader view of negotiation: maximizing the quality of service provided to the most critical applications while minimizing the impact on other applications. The QuO project [7][8] terms the adaptation of object methods to the load as negotiation. Adaptation is only one aspect of QoS/resource

management in DeSiDeRaTa, with dynamic optimization of system resource utilization and application QoS being other capabilities of DeSiDeRaTa’s QoS negotiation. The University of Colorado DQM’s [9][10] negotiation concept is a means of raising and lowering the operating level (the algorithm’s complexity) based on current CPU usage conditions. This use of the term “negotiation” is similar to QuO’s use of the term. EPIQ’s [11] description of negotiation falls under this description as well, with the switching of regions of feasible quality being done in response to current conditions. The RTPOOL project [12] describes negotiation as the client’s specifying a static deadline for a task with a reward for scheduling the task. The server does a preliminary static schedulability analysis of worst-case timing characteristics, and its algorithm shuffles the tasks to maximize the reward. DeSiDeRaTa is a dynamic system that maintains the required quality of service under dynamic workloads, where worst-case execution times and time-invariant statistical timing characteristics are unknown. Furthermore, it uses the path abstraction.

6 Conclusions and Future Work Algorithms have been developed that will allow middleware to negotiate for the highest possible quality of service for distributed, dynamic real-time systems. The path abstraction allows QoS management to be decentralized and provides the basis for negotiating for resources for applications of differing criticality and purpose. The supply and demand approach to QoS management is based on the concept that resources (supply space) are limited in quantity and capacity and that the paths’ applications are the consumers (demand space) of these resources. If the applications cannot have their desired amounts of resources, then the middleware needs to distribute resources in order to deliver the best QoS possible. The major contributions of the paper are the negotiation algorithms and protocol that minimize the impact on the other paths’ QoS while maximizing the unhealthy path’s QoS. Future work includes implementation of the negotiation algorithms and integration into the current QoS and resource managers.

References 1. Tsai, J. J. P., and S. J. H. Yang. Monitoring and Debugging of Distributed Real-Time Systems. Los Alamitos, CA: IEEE Computer Society Press, 1995. 2. Cavanaugh, C. D., L. R. Welch, and C. Bruggeman. A Path-Based Design for the Air Traffic Control Problem. Arlington, TX: The University of Texas at Arlington Department of Computer Science and Engineering, 1999. Technical Report, TR-CSE-99-001. 3. Harrison, R. D. "Combat System Prerequisites on Supercomputer Performance Analysis." Proceedings of the NATO Advanced Study Institute on Real Time Computing, 1994. 4. Welch, L. R., B. Ravindran, B. Shirazi, and C. Bruggeman. "Specification and Analysis of Dynamic, Distributed Real-Time Systems." Proceedings of the 19th IEEE Real-Time Systems Symposium, Madrid, Spain, December 2-4, 1998. 5. Welch, L. R., P. V. Werme, B. Ravindran, L. A. Fontenot, M. W. Masters, D. W. Mills, and B. A. Shirazi. "Adaptive QoS and Resource Management Using A Posteriori Workload

Characterizations." Proceedings of the 5th IEEE Real-Time Technology and Applications Symposium (RTAS '99), May 1999. 6. Welch, L. R., and B. A. Shirazi. "A Dynamic Real-time Benchmark for Assessment of QoS and Resource Management Technology." Proceedings of the 5th IEEE Real-Time Technology and Applications Symposium (RTAS '99), May 1999. 7. Loyall, J. P., R. E. Schantz, J. A. Zinky, and D. E. Bakken. "Specifying and Measuring Quality of Service in Distributed Object Systems." Proceedings of the 1st International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC '98), Kyoto, Japan, April 1998. 8. Zinky, J. A., D. E. Bakken, and R. E. Schantz. “Architectural Support for Quality of Service for CORBA Objects”, Theory and Practice of Object Systems, 3(1) 1997. 9. Brandt, S., G. Nutt, T. Berk, and J. Mankovich, "A Dynamic Quality of Service Middleware Agent for Mediating Application Resource Usage", Proceedings of the 19th IEEE RealTime Systems Symposium (RTSS '98), December 1998. 10. Brandt., S., G. Nutt, T. Berk, and M. Humphrey, "Soft Real-Time Application Execution with Dynamic Quality of Service Assurance", Proceedings of the 6th IEEE/IFIP International Workshop on Quality of Service (IWQoS '98), pp. 154-163, May 1998. 11. Liu, J. W. S., K. Nahrstedt, D. Hull, S. Chen, and B. Li. “EPIQ QoS Characterization Draft Version.” http://epiq.cs.uiuc.edu/qo-970722.pdf 12. Abdelzaher, T. F., E. M. Atkins, and K. Shin, “QoS Negotiation in Real-Time Systems and its Application to Automated Flight Control” accepted to IEEE Transactions on Software Engineering, 1999. (Earlier version appeared in IEEE Real-Time Technology and Applications Symposium, Montreal, Canada, June 9-11, 1997.