Optimized Priority Assignment for Tasks and Messages in ... - CiteSeerX

Optimized Priority Assignment for Tasks and Messages in Distributed Hard Real-Time Systems

By: José Javier Gutiérrez García, and Michael González Harbour Departamento de Electrónica, Universidad de Cantabria, SPAIN

Abstract Recent advances in the analysis of distributed realtime systems have made it possible to predict if hard realtime requirements will be met. However, it is still difficult to find a feasible priority assignment when the utilization levels of the CPUs and communication networks are pushed near to their limits. This paper presents a new heuristic algorithm for optimizing the assignment of priorities to tasks and messages in distributed hard realtime systems. The algorithm is based on the knowledge of the parameters that influence the worst-case response time of a distributed application. This algorithm is compared to simulated annealing, which is a general optimization technique for discrete functions that had been previously used for solving similar problems. On average, our heuristic algorithm executes two orders of magnitude faster than simulated annealing, finds better solutions, and finds solutions in cases where the latter method fails.

1. Introduction Distributed real-time systems have an increasing importance in today’s control systems, since low-cost networking facilities allow the interconnection of multiple devices and their controllers into a single large system. The typical architecture in a distributed real-time control system consists of several processor nodes interconnected through one or more interconnection networks. The system’s software is composed of concurrent tasks that are often statically allocated to processing nodes because the effects of dynamic node allocation are very difficult to _______________________ This work has been supported in part by the Comisión Interministerial de Ciencia y Tecnología of the Spanish Government, under Grant ROB91-0288

predict for hard real-time systems. Each task may exchange messages with other tasks in the same node or in different nodes. Messages exchanged among tasks in different nodes are also statically allocated to a particular network. Tasks allocated in the same node may also share data or resources via the usual synchronization mechanisms used in shared memory systems. Each task in the system is activated by the arrival of a triggering event that may be generated by the external environment, a timer, or by the arrival of a message from another task. Many of these events have an associated timing requirement, such as an end-to-end deadline, which is the maximum response time allowed from the arrival of an event to its final completion. During the execution of the response to this event, several tasks may need to be executed in one or more nodes, and several messages exchanged in the network. Other timing requirements may also be imposed on individual actions. In order to make it possible to predict worst-case response times, it is possible to schedule the activities in each processing node using a priority-based approach that allows processing activities with shorter deadlines before (at a higher priority) activities with longer deadlines. One analysis technique used to assess the schedulability of the real-time system is called Rate Monotonic Analysis (RMA) and is well defined for single-processor systems [4]. The RMA techniques can also be applied to distributed systems [4][7][10] by modeling each network as if it were a processor, and each message as if it were a task. Several problems appear in distributed systems that have to be considered. Two of the most important are the task and message allocation, and the assignment of priorities. Tindell et al [9] describe a solution that can be used to solve the first two issues by using a discrete

optimization algorithm (simulated annealing) to find a close-to-optimum solution to this problem. The work that is presented in this paper represents an optimization of existing techniques for the assignment of priorities in distributed hard real-time systems. The paper is organized as follows. In Section 2 we provide a quick review of the model that we use for the distributed system, and the analysis method that we use. Section 3 describes a new heuristic algorithm for optimizing the assignment of priorities in distributed real-time systems. In Section 4 we compare the results of our algorithm with those of simulated annealing, showing that our algorithm finds better solutions in much less time. Finally, in Section 5 we give our conclusions.

2. System Model and Current Analysis Techniques In a real-time distributed system there are multiple processing resources (CPUs) and one or more communication resources (backplane buses, LANs, serial lines, etc.). The problem of scheduling and analyzing the response time of concurrent programs running in the processing resources is well known. We will assume fixed-priority preemptive scheduling, and we will use Rate Monotonic Analysis (RMA) to analyze the worst-case timing behavior. Scheduling and analyzing the worst-case response time in the communication resources is not so well established. Most network standards are not well suited to real-time communications and do not support prioritybased scheduling of messages. However, there are some network standards that have been used to accomplish hard real-time communications, such as the token-bus [8], FDDI [1], etc. Also some non-standard networks have been developed to accomplish preemptive priority-based scheduling. At the University of Cantabria we have developed a software tool that allows us to accomplish preemptive priority-based scheduling over standard communication subsystems such as the VME bus, and point-to-point serial lines (RS232, RS485) [3]. The analysis of message traffic on these communication resources is carried out using the same RMA techniques that are used in CPUs. As a consequence, in our distributed system we treat messages and communication resources exactly as if they were tasks in processing resources, except for a small amount of non preemption

that must be taken into account because messages can only be preempted at packet boundaries. In our hard real-time system model we assume that all event sequences that arrive at the system are known in advance, and that the worst-case rates of those events with hard real-time requirements are also know. Otherwise, it would be impossible to guarantee the schedulability of the system in advance, at compile time. We also assume that tasks are statically assigned to processors, and similarly messages to communication networks. In many hard realtime systems tasks are tied to specific processors because of the presence of special hardware devices that are needed by the tasks. Each event ei that arrives at the system generates a response in form of a sequence of actions (ai1 ain). Each of these actions may be a task (or a task portion) executed in a processor, or a message sent across a network. Usually the first action of a response is a task running in some particular processor. This task may either activate another task in a different processor, or a message that, in turn, activates another task on another processor. The timing requirements that we consider are global deadlines that start at the arrival of the event, and must be met by particular actions in the sequence. We call them global deadlines to distinguish them from local deadlines which are relative to the activation of a particular action in the response. We will call Dij the global deadline of action aij (relative to the arrival of the event) and dij the local deadline of action aij (relative to the activation of aij). The global deadline corresponding to the last action in the response is called the end-to-end deadline (Di). This model described here is well-suited to represent a large number of real-time architectures that are found in practice. For example, if the system is using a clientserver approach, each portion of a task that invokes a service from a remote server may be decomposed into the following actions: the action before invoking the service, the message sent to the server, the server’s action, the reply message, and the action after invoking the server. Figure 1 shows an example of such a decomposition, in a distributed system with two nodes and one network. Our model, however, does not allow tasks to synchronize, or to be activated from combinations of events. Work to eliminate these restrictions is currently under way. The general worst-case analysis of a distributed system is still an open issue, but one approach that can be used is to analyze each processing and communication resource in the system as if they were independent. This

CPU-1

NETWORK

CPU-2

(relative to the arrival of the event) as the sum of the local response times of itself and of all the previous actions: j

Task 1

Server 1

Task 2

Server 2

Task 1_1

M1_1

Task 1_2

M1_2

Task 2_1

M2_1

Task 2_2

Server 1

Server 2

M2_2

Figure 1. Modelling tasks that invoke remote servers

approach is pessimistic, in the sense that the actual worst case may take advantage of the fact that the phases or timing relationships between actions in different responses are interrelated. By considering the phase of each action as independent, we may build a situation that is worse than the worst case. However, although slightly pessimistic, this method allows us to guarantee the schedulability of real-time systems. The analysis of a system like the one described is carried out using the normal RMA techniques, but taking into account the effect of jitter [10]. Since messages are activated by the execution of tasks, their activation time is not perfectly periodic, but depends on the completion time of the triggering task, which is variable. The same happens with tasks activated by the arrival of a message. The solution to this issue is addressed by Tindell [10] by including the effects of deferred activation (also called jitter) in the schedulability analysis. For each event ei we define its period Ti as the inverse of the worst-case rate at which that event may arrive at the system. For each action aij in the response to event ei we define the global worst-case response time Rij

Rij

rik k 1

where rik is the local response time of action aik (relative to the activation of aik). In the calculation of the local response times we must assume a worst-case jitter Jik equal to the variation of the response time of the previous action. We will assume that this variation is equal to the global worst-case response time of the previous action (Rik-1) because, in general, the best-case response time could be arbitrarily small. We will call Ri the global worst-case response time of the last action in the response. Although the analysis is recursive (because the response time depends on the jitter and the jitter depends on the response time) Tindell [10] described how the analysis could be carried out in an iterative way until a stable solution for the jitter terms and the response times was reached. Note that to determine the schedulability of the system, the global response times (Ri’s) must be compared with the global deadlines (Di’s).

3. Heuristic Algorithm Priority Assignment

for

Optimized

The solution to the problem of optimizing the assignment of priorities is well known for uniprocessors. Liu and Layland [6] proved the optimallity of rate monotonic priorities for sets of independent periodic tasks with the deadline of each task equal to its period. Leung and Whitehead [5] proved the optimallity of deadline monotonic priorities for the case when deadlines were less than or equal to the periods. Audsley [2] provided an algorithm that was capable of finding a solution (if one existed) to the problem of assigning priorities when one or more of the deadlines were larger than the task periods. However, in multiprocessor or distributed systems the problem is much more complicated, because of the strong interaction between the response times of actions in the different resources. Tindell [9] treated the problems of priority assignment and task allocation combined, as a discrete optimization problem, and used simulated annealing as a suitable optimization technique. In this paper we propose a new heuristic algorithm, HOPA (heuristic optimized priority assignment) that takes advantage of the knowledge of the factors that influence

the timing behavior to find an optimized solution to the priority assignment problem (as opposed to the simulated annealing technique which is general purpose). The HOPA algorithm is based on the distribution of the end-to-end or global deadlines of each response among the different actions that compose that response. Once each action is assigned an artificial local deadline, deadline monotonic priorities are assigned in each resource and an analysis of the whole system is carried out. As a result of the analysis, new intermediate deadlines are calculated. The iteration proceeds until a suitable solution is found or some stopping condition is reached.

algorithm HOPA is begin assign initial local deadlines; loop assign priorities according to deadlines; calculate worst-case response times; exit when some stopping criterion; calculate new local deadlines; end loop; end HOPA;

The initial local deadlines can be assigned in any way that preserves the end-to-end deadline (i.e., Di = ∑ dik, for all actions aik in the response). One possible assignment is to distribute the end-to-end deadline proportionally to the worst-case execution time of each action (Cik). Once the local deadlines are known for each action, the assignment of priorities is carried out independently in each resource, using the deadline monotonic priority assignment, that is, assigning the highest priority to the action with the shortest deadline, and so on. After all the priorities have been assigned, the system is analyzed using the technique described in [10] that was mentioned in Section 2. The analysis provides the worst-case response times for each action, for that priority assignment. If the solution is suitable (or some other stopping condition has been reached) the algorithm terminates. Otherwise, new local deadlines are calculated by redistributing the global deadlines among the actions of a response. This redistribution takes into account the “excess” of each action which, intuitively, measures the distance that separates each action from schedulability. The new intermediate deadlines are obtained as a function of two

factors: on the one hand, the excess of each action relative to other actions in the same response; on the other hand, the excess of each action relative to the other actions in the same resource. For each action aij we define the “excess” as either the “excess of response time” :

exc(aij)

  Ri   (rij dij) Di     (r J d ) Ri  ij ij ij D i  

; when dij ≤ Ti

; when dij > Ti

or the “excess of compute time”:

exc(aij)

  Ri  ; when dij ≤ Ti  ( sij) D  i    (r J d ) Ri ; when d > T ij i  ij ij ij D i  

where sij is the slack time of action aij, which is defined as the amount of execution time (with negative sign) that must be subtracted from action aij for it to meet its local deadline dij, if aij is not schedulable, or the amount of execution time that may be added to action aij for it to continue being schedulable, if it is schedulable. Notice that the excess may be negative, indicating that the associated action meets its timing requirements comfortably. The reason for having two different definitions for the “excess” of a particular action aij is that they make the HOPA algorithm behave differently. The first definition, excess of response time, leads to faster solutions because calculating the slack time is a complex operation. The second definition leads to better solutions because, for deadlines smaller than periods, the slack time is a much better representation of the excess. In the HOPA algorithm we start using the first definition, and after a number of iterations without finding a solution, we try the second one. The reason for distinguishing the cases when dij>Ti or dij≤Ti is because when the local deadlines are larger than the periods, jitter has a strong influence (not on the response time of action aij itself, but on lower priority tasks). For deadlines that are less than or equal to the

periods, the impact of jitter is not so strong and is less important than the excess of slack or of response time. The excess equations can be modified to support the case in which particular actions have global deadlines other than the final end-to-end deadline. In this case, instead of Ri, we use the difference between the global response times of the actions before and after aij that have a global deadline. In a similar way, we use the difference between those global deadlines, instead of Di. For each processing or communications resource PRk we define the “excess” in that resource as: exc(PRk)

aij∈PRk

exc(aij)

In addition, we define the maximum excess of all the resources, and the maximum excess of all the actions belonging to a particular response as: Mex(PR)

max exc(PRi)

Mex(ei)

max exc(aij)

∀ PRi

 exc(PRk)  dij(old) 1   kR Mex(PR) 

 exc(aij)  1  k Mex(e )  a i  

where kR and ka are constants that control the relative influence of the resource and action components in the calculation of the new deadline. The smaller these constants are, the higher this influence is. Appropriate values for these constants are between 1.5 and 3.0. In the HOPA algorithm we start with values of kR=ka=2.0, and we then change the values to 1.8, 3.0, 1.5, etc., until we reach a feasible solution. Once the new local deadlines have been obtained for all actions of a response, we adjust them proportionally to make them fit into the global deadlines. For an end-to-end deadline we do:

dij(new) ∀k

dik

Di

As a stopping criteria for the HOPA algorithm we limit the maximum number of iterations, and we stop whenever two consecutive deadline assignments are identical (in which case the algorithm would continue providing the same solution), or when a suitable solution is found. After the algorithm finds a priority assignment that makes the system schedulable, it is capable of finding a better optimized solution by executing more iterations. To determine how good a solution is compared to others, we define a scheduling index as a function of the worst distance between the worst-case response times and the global deadlines for each response:

∀j

Given these definitions of excess times, we calculate the new local deadline for each action as a function of the old local deadline for that action, the excess for that action, and the excess for the resource to which that action belongs, as:

dij(new)

dij

Index

        

∀i

∀i

min (0,Di Ri) ; when not schedulable

max (0,Di Ri) ; when schedulable

For each intermediate action with a global deadline, a similar term is included in the above equations for calculating the minimum or the maximum, using the global deadline minus the global response time for that action. Negative values of the scheduling index represent solutions that make the system unschedulable, and positive or zero values of the index represent schedulable solutions. The larger the index becomes, the better the schedulability of the system is in terms of distance between the response times and the deadlines.

4. Comparison of the Heuristic Algorithm with Simulated Annealing In this section we compare the performance of our heuristic algorithm with the simulated annealing technique. Simulated annealing is a global optimization technique that attempts to find the lowest point in an energy landscape. It has been successfully used for many discrete

one resource (CPU or network), selecting two actions (tasks or messages), and exchanging their priorities. As a comparison platform, we will use an example of a distributed system with eight processors interconnected via three networks (Figure 2). Any processor can send messages to any other processor via any of the networks. The system must respond to seven event sequences. Each event generates a sequence of tasks and messages that are shown in Figure 3. The assignment of tasks to processors and messages to networks are shown in Table 1. We define the following timing requirements for the events of this system.

Proccessor 3 Proccessor 4

Proccessor 2

Network 1

Proccessor 1

Network 2

Network 3

Proccessor 8

Proccessor 5

Proccessor 6 Proccessor 7

Figure 2. Example of a distributed real-time system optimization problems. Tindell et al [9] used it to assign priorities and allocate tasks to processors in hard real-time distributed systems. We have compared the performance of our heuristic algorithm with simulated annealing, although we have redefined the energy function to allow the annealing algorithm to optimize over an already schedulable solution. The energy function that we used was the negative of the scheduling index that was defined above, and the neighbor function consisted of choosing

E1 E2

m1,2 t1,1

m1,4 t1,3

m2,2 t2,1

E3

m2,4

m3,2

E4

t3,3

t4,1

m4,4 t4,3

m5,2 t5,1

E6

t5,3 m6,2

t6,1

E7

m6,4 t6,3

m7,2 t7,1

t6,5

t6,7

t3,13

t4,11

t3,15 m4,14

t4,13 m5,12

t5,11

t6,9 m7,8

m7,6 t7,5

t5,9

m3,14

m4,12

m5,10

t2,15

t4,15 m5,14

t5,13

t5,15

m6,8

m6,6

m7,4 t7,3

t5,7

t2,13

t3,11

t4,9

t1,15 m2,14

m3,12

m4,10

m5,8

t1,13

t2,11

t3,9

t4,7

t5,5

4000 5000 7000 4500 5500 3500 5000

m1,14

m2,12

m3,10

m4,8

m5,6

= = = = = = =

m1,12 t1,11

t2,9

t3,7

t4,5

D1 D2 D3 D4 D5 D6 D7

400 500 700 450 550 350 500

m2,10

m3,8

m4,6

m5,4

t1,9

t2,7

t3,5

= = = = = = =

m1,10

m2,8

m3,6

T1 T2 T3 T4 T5 T6 T7

We will now apply both our heuristic algorithm and simulated annealing to the problem of assigning priorities in this system. For this purpose, we have implemented a

t1,7

t2,5

--> --> --> --> --> --> -->

m1,8

m2,6

m3,4

m4,2

E5

t1,5

t2,3

t3,1

m1,6

E1 E2 E3 E4 E5 E6 E7

t7,7

ti,j mi,j

t7,9

Figure 3. Structure of the event sequences in the example

Task Message

Figure 4. Comparison of the heuristic algorithm with simulated annealing software tool, written in Ada, that can apply both methods using a parametric description of the real-time system. The tool allows us to gradually increment the work load in one or more of the resources, in order to reach the limit where any of the algorithms cannot provide schedulable solutions. Just to realize how large the problem is, we will mention that the number of possible priority assignments in this problem is in the order of 1058.

Processor Processor Processor Processor Processor Processor Processor Processor Network 1 Network 2 Network 3

1 2 3 4 5 6 7 8

: : : : : : : : : : :

Figure 4 shows a comparison of the performance of both algorithms using the example described above. The range of resource utilizations has been chosen close to the limit where both algorithms fail to find a feasible solution. The graph shows for each algorithm both the execution time, and the scheduling index. Recall that the scheduling index represents the optimallity of the solution in terms of the worst distance between the deadlines and the

t1,1,t2,1,t3,5,t4,1,t5,15,t6,1,t7,7 t1,3,t2,5,t3,7,t4,7,t5,13,t6,9,t7,9 t1,5,t2,3,t3,3,t4,15,t5,11,t7,1 t1,7,t2,9,t3,1,t4,13,t5,9,t7,5 t1,9,t2,7,t3,15,t4,5,t5,7,t6,3 t1,11,t2,13,t3,13,t4,3,t5,5,t6,5 t1,13,t2,11,t3,11,t4,9,t5,3,t6,7 t1,15,t2,15,t3,9,t4,11,t5,1,t7,3 m1,2,m1,4,m1,6,m1,8,m1,10,m1,12,m1,14,m2,2,m2,4,m2,6,m2,8,m2,10,m2,12,m2,14 m3,2,m3,4,m3,6,m3,8,m3,10,m3,12,m3,14,m4,2,m4,4,m4,6,m4,8,m4,10,m4,12,m4,14 m5,2,m5,4,m5,6,m5,8,m5,10,m5,12,m5,14,m6,2,m6,4,m6,6,m6,8,m7,2,m7,4,m7,6,m7,8

Table 1. Assignment of tasks to processors and messages to networks

corresponding worst-case response times. Negative scheduling indexes represent failure to find a schedulable solution. We noticed that in many of the cases where simulated annealing was not able to find a solution, our heuristic algorithm was able to find at least one. Since the solution found by simulated annealing may depend on the initial guess for the priority assignment, we repeated the algorithm several times to give it a chance to find a solution (although at the expense of more processing time). Two important results can be derived from this graph. First, the heuristic algorithm finds more and better solutions than simulated annealing. Second, and also important, the execution time of our algorithm is, on average, about two orders of magnitude less than the time spent by simulated annealing. As an example, the total CPU time consumed by the two algorithms to generate the points in this graph were: Heuristic Algorithm : 492.3 s. Annealing Algorithm : 425284.0 s. The analysis tool has been applied to many more examples with very similar results. In the comparison of the two methods, we noticed that for low utilizations both methods find a feasible solution very quickly, although the heuristic algorithm finds better solutions. However, when the utilization of the resources is high (and thus the problem is harder), the difference in performance between the heuristic algorithm and simulated annealing is enormous. This difference in favor of the heuristic algorithm is even larger when the deadlines are larger than the task periods. This is a very common situation in distributed systems.

that it finds feasible solutions in many cases where simulated annealing fails. This is especially noticeable when utilizations are high or when the deadlines of the responses are larger than the periods of the events that activate them. Given the promising performance of the proposed heuristic algorithm, we plan to apply it to other problems. We are currently working in the analysis of more complex systems, such as systems in which actions of the different responses synchronize to share resources, or interact with other actions in more sophisticated ways, for example being activated by combinations of events. We also plan to eliminate the current restriction that forces consecutive actions on the same resource to be collapsed into a single action, and thus share the same priority. We will also try to find a similar algorithm to solve the problem of allocating tasks to processors and messages to communication networks.

Acknowledgments We would like to thank the referees of this paper for the valuable comments and inputs.

References [1]

[2]

[3]

5. Conclusions In this paper we propose a heuristic algorithm for assigning priorities in distributed hard real-time systems. We have shown that this method can find feasible optimized solutions in a small amount of time, even in situations where the utilization of the resources is high and thus the number of solutions is very small. We have compared this algorithm with simulated annealing, which is a general purpose discrete optimization algorithm, that was previously used to address the same problem. The results of the comparison are that our method is two orders of magnitude faster on average, and

[4]

[5]

[6]

[7]

G. Agrawal, B. Chen, W. Zhao, and S. Davari, "Architecture Impact of FDDI Network on Scheduling Hard Real-Time Traffic". Workshop on Architectural Aspects of Real-Time Systems, December 1991. N.C. Audsley, "Optimal Priority Assignment and Feasibility of Static Priority Tasks with Arbitrary Start Times", Dept. of Computer Science, Report no. YCS 164, University of York, December 1991. J.J. Gutiérrez García, "Planificación y Optimización de Mecanismos de Paso de Mensajes para Sistemas Distribuidos de Tiempo Real Estricto", PhD Thesis, University of Cantabria, in preparation. M. Klein, T. Ralya, B. Pollak, R. Obenza, and M. González Harbour, "A Practitioner’s Handbook for RealTime Systems Analysis". Kluwer Academic Pub., 1993. J.Y.T. Leung, J. Whitehead, "On the Complexity of Fixed-Priority Scheduling of Periodic Real-Time Tasks", Performance Evaluation, 2, (4), Dec. 1982, pp.237-250. C.L. Liu, and J.W. Layland, "Scheduling Algorithms for Multiprogramming in a Hard Real-Time Environment". Journal of the ACM, 20 (1), 1973, pp 46-61. L. Sha, and S.S. Sathaye, "A Systematic Approach to Designing Distributed Real-Time Systems". IEEE Computer, September 1993, pp. 68-78.

[8]

[9]

J.K. Strosnider, T. Marchock, J. Lehoczky, "Advanced Real-Time Scheduling Using the IEEE 802.5 Token Ring". Proceedings of the IEEE Real-Time Systems Symposium, 1988. K.W. Tindell, A. Burns, and A.J. Wellings, "Allocating Real-Time Tasks. An NP-Hard Problem Made Easy". Real-Time Systems Journal, Vol.4, No.2, May 1992.

[10]

K. Tindell, and J. Clark, "Holistic Schedulability Analysis for Distributed Hard Real-Time Systems". Microprocessing & Microprogramming, Vol. 50, Nos.2-3, pp. 117-134, April 1994.