Empirical Quantification of Pessimism in State-of- the-Art Scheduling Theory Techniques for. Periodic and Sporadic DRE Tasks. Gautam H. Thaker, Patrick J.
Empirical Quantification of Pessimism in State-ofthe-Art Scheduling Theory Techniques for Periodic and Sporadic DRE Tasks Gautam H. Thaker, Patrick J. Lardieri, Dr. Donald K. Krecker, and Michael Price Lockheed Martin Advanced Technology Laboratories Cherry Hill, NJ 08002 {gthaker, plardier, dkrecker, mprice}@atl.lmco.com Abstract—Distributed, Real-time, Embedded (DRE) systems present numerous challenges with respect to certification of their real-time behavior. Ideally, to address these we would like to build a model of our system that captures relevant information about end to end real-time requirements, resource consumptions requirements and resource availability, and subject the model to real-time scheduling analysis to predict performance. Presently, scheduling theory techniques have seen limited application in DRE systems for multiple reasons including pessimistic predictions of worst-case response times. Our study quantifies the pessimism in the predictions of worst-case response times of competing end-to-end distributed periodic tasks by comparing values observed in simulation with values computed using multiple scheduling theory techniques. Specifically we consider non-greedy synchronization protocols for tasks with a high degree of recurrence. Our results show that for an end-to-end task model non-greedy techniques, when used with proportional deadline monotonic scheduling, reduce the pessimism in worst-case response time predictions to within 5% of the actual value in over 90% of cases. These (quasi) static techniques represent a baseline against which we can evaluate emerging, control theoretic, adaptive scheduling methods Keywords—real-time scheduling, scheduability analysis, periodic tasks, distributed task scheduling, end-to-end task scheduling
INTRODUCTION Distributed scheduling theory has been researched for many years and numerous analysis techniques exist. However, use of those techniques has been limited. The following reasons are often cited for this: 1. Distributed Scheduling techniques are generally limited to a (mostly static) collection of periodic processes and this model does not fit all real systems. 2. It is difficult to obtain accurate values for parameters that specify such a system. In particular, this applies to estimates for worst-case execution of subtasks. As a result highly pessimistic values are used, which leads to pessimistic end-to-end response time predictions. 3. Even when accurate execution times and other parameters can be obtained, the distributed scheduling analysis is
fundamentally pessimistic and leads to vast over-estimates of worst-case end-to-end response times. This pessimism renders this approach useless1. 4. Support is lacking at the operating system, network, and crucial middleware layers to successfully build DRE systems with hard real-time requirements. This study is primarily concerned with addressing the third issue, which we believe is the most serious impediment to applying scheduling theory techniques to DRE systems. However, as all these issues must be addressed to successfully apply scheduling theory techniques to DRE systems we first briefly comment on items 1, 2 and 4 from the above list. 1. Limited Applicability of the Periodic Task Model – The periodic task model with extensions for sporadic tasks can be applied to many distributed system workflows in real systems. For example, in the Navy’s 21st Century Destroyer DD(X), navigation data, radar track data and other sensor data drive a periodic track correlation process that is distributed over multiple processors. The impediment to applying scheduling theory in these cases has been pessimism (which leads to low utilizations) and not doubling the periodic task model. The scheduling techniques in this paper address this pessimism issue. There are also newly emerging techniques [1] that address the schedulability of aperiodic tasks in isolation and in combination with periodic tasks. We plan to perform similar evaluations of those and other techniques in the future. 2. Inaccessibility of True Task Performance Parameters – Very limited analytical techniques exist for quantifying the worst-case execution time of a task (a self contained unit of executable code) and these values are typically obtained through experimentation. Statistical distributions for task execution times are developed using a large sample sizes from a diverse input population. Using this approach we can compute the mean and variance of the distribution but
1
“Model Checking” represents an alternate technique to distributed scheduling theory for estimating worst case end to end response times. However, most model checking is based on some form of exhaustive state search and is generally suitable for very small systems at best. Model checking should, in theory, produce “true” (non-pessimistic) worst case estimates.
we still do not know the probability distribution function. However, with respect to scheduling theory this is problematic only in cases where the variance in task execution times is large. In cases where variance is low we have utilized jitter analysis techniques to determine the sensitivity of the scheduling theory predictions to minor variations. In cases where variance is large, scheduling theory still applies. However, we typically accept low average processor utilizations to meet our hard real-time requirements. Future work in statistical scheduling techniques may some day address this problem. 3. COTS Infrastructure Unsuitable for Real-Time Systems – The lack of quality of service (QoS) mechanism in commercial mainstream technologies has been overcome in recent years. The new QoS mechanisms permit system designers the necessary control to build DRE systems upon COTS infrastructure [2]. Our purpose for this study is two fold. One, we hope to establish the feasibility and value of applying periodic scheduling techniques as part of the performance engineering process for next generation military DRE systems. Two, these advanced static scheduling analysis techniques that remove significant pessimism form a performance baseline to compare the value of emerging dynamic and adaptive scheduling techniques. The remainder of the paper presents our work to quantify the pessimism in scheduling analysis of distributed end-to-end periodic tasks. The paper is organized as follows: Section 2 presents our experimental methodology; Section 3 defines the specific experiments we undertook; Section 4 presents the results obtained for each experiment; Section 5 discusses conclusions drawn the entire study; and Section 6 presents our plans for future work.
EXPERIMENTAL METHODOLOGY In considering pessimism we first distinguish different measures that can be considered for worst-case end-to-end response time (WCRT) for tasks. 1. True WCRT that is likely to occur. This value may only be known to an “all knowing oracle". 2. WCRT bounded by scheduling analysis. This value is an upper bound on WCRT and should never be smaller than "true" WCRT. 3. WCRT observed in a running system or in a detailed, discrete event simulation model of the system. In both of these cases the system must “run” for some adequate amount of time under the entire spectrum of conditions it is likely to encounter. In either case it is impossible to know if the conditions leading to "true" WCRT have been encountered. Thus, values obtained are a lower bound on WCRT. Lower bounds obtained from measurements of a running system or a simulation may be equal to each other or either may be larger than the other. However, neither should be larger than the upper bound mentioned above. In this paper we evaluate a large number of different systems on a 24 node network. Due to the difficulties of
building such a large testbed2 we rely on detailed simulations to compute the lower bound on WCRT. We have previously shown [3] that for a three node, nine subtask system simulations and actual systems are generally in good agreement. We quantify pessimism by measuring the deviation or gap between the WCRT predicted by scheduling analysis (the upper bound) and the WCRT observed in simulation (the lower bound). The true WCRT lies between these bounds. The degree of pessimism is directly proportional to the gap size. In our study we considered tasks comprised of linear chains of subtasks distributed across a network. We typically refer to these as end-to-end tasks or just tasks. When performing scheduling analysis on tasks, various properties of the system and our scheduling decisions affect pessimism. We have studied the effects of varying two3 scheduling decisions, subtask prioritization and subtask synchronization, and one system property, subtask recurrence. These are briefly defined in Table 1. We employed the following methodology in our study. 1. Following on recent work [4] we examine how various subtask prioritizations and subtask synchronization protocols affects the predicted WCRT across a large set of randomized tasks. Specifically we are looking for a combination of subtask prioritization and subtask synchronization protocol that consistently provides the tightest upper bound (i.e., the smallest WCRT). 2. For techniques with the tightest upper bounds, we develop discrete event simulations that execute the same task set under the corresponding subtask prioritization and subtask synchronization protocols and record (among other things) the observed WCRT. Using these results we quantify the pessimism in the analytical techniques. TABLE 1. TASK PROPERTIES THAT EFFECT PESSIMISM Scheduling Decision/ Task Property Subtask Prioritization
Subtask Synchronization Subtask Recurrence
Description Subtask Prioritization determines the strategy for mapping priorities to subtasks. On a given processor subtasks are executed using preemption based on priorities. Subtask Synchronization determines the timing protocol for release of non-initial subtasks in an end-to-end task. Subtask Recurrence measures the degree to which the subtasks of a particular task can interfere with each other’s execution on a specific processor. Once subtasks are allocated to processors this value is fixed. In our experiments we randomly assign subtasks to processors and subsequently do not change this assignment. Thus, subtask recurrence is a property of each of our test systems.
2 In future we plan to use a facility such as www.emulab.net for our physical system tests. 3 We have also informally studied the effects of jitter in subtask execution times and task periods. Those results are not documented in this paper.
3. Using innovations developed at Lockheed Martin Advanced Technology Laboratories (ATL)4 we examine how subtask recurrence affects pessimism in predicted WCRT. (Recurrence means task has more than one subtask on a single processor.) Specifically we attempt to determine the degree to which ATL’s “Recurrence Aware Analysis” reduces pessimism of the base techniques.
EXPERIMENTAL PLAN For our experiments, we generated 50 randomized task sets or systems from the parameters summarized in Table 2. These specific values generally reflect the likely configuration present in a single computing cluster of a next generation Navy Surface Combatant. Each system contains 12 tasks and 48 subtasks that are randomly assigned to processors. An experiment operates on all 50 systems at 75 discrete processor-load levels (25%-99%). Particular values of processor-load are achieved by scaling the worst-case execution times of the set of subtasks assigned to the each processor. This produces 45,000 worst-case response time (WCRT) samples per experiment5 as computed below: 50 systems x 12 tasks/system x 75 different CPU Loadings = 45,000 WCRT samples/experiment We define an experiment by specifying a subtask prioritization and a subtask synchronization protocol. Additionally we specify whether the results are analytical or simulated. For analytical results we note if subtask recurrence was considered in the analysis. The experiments we performed in our study are summarized in Table 3. The analytical experiments associated with the first step in our methodology are similar to those performed by [4] and provide a base performance level for our evaluation. The simulated experiments enable us to quantify the pessimism in these techniques; a study that Sun did not undertake formally. TABLE 2. RANDOMIZED TASK SET PARAMETERS Parameter
Value
Number of Tasks Subtasks per Task
12 4
Task Period
Exponential Distribution (Mean=1000, Min=100, Max=10000)
Task Deadline Subtask Worst-Case Execution Time
4*Period Uniform Distribution (Min=0, Max=1)
Initial Task Phase
Uniform Distribution (Min=0,Max=Task Period)
Number of Processors Processor Loading
24 Monotonically increasing from 25%-99% in increments of 1% (achieved by scaling Subtask Worst-Case Execution Times)
TABLE 3. EXPERIMENTS PERFORMED Experiment Designator DS_RM
DS_PD
PM_RM
The theoretical details of these innovations are beyond the scope of this paper. we hope to publish them in a separate report 5 In some experiments there may be less than 45,000 WCRT values because the analysis failed to converge in all cases.
Subtask Prioritization
Direct Synchronization (DS) DS
Rate Monotonic (RM) Proportional Deadline Monotonic (PD) RM
Recurrence Considered No
Result Typea
Method Step
Analytical
1
No
Analytical
1
No
Analytical
1
PD
No
Analytical Simulated Simulated
1
Simulated Simulated Analytical Analytical
2
PM_PD
Phase Modification (PM) PM
DS_RM_s
DS
RM
N/A
RG_RM_s
RM
N/A
DS_PD_s
Release Guardb (RG) DS
PD
N/A
RG_PD_s
RG
PD
N/A
R_PM_RM
PM
RM
Yes
R_PM_PD
PM
PD
Yes
2 2
2 3 3
a The equations for the DS_* and the PM_* techniques are detailed in [4]. The equations for the R_PM_* extensions are available from the authors. b The Release Guard (RG) Synchronization Protocol is a non-greedy synchronization technique that is closely related to phase modification technique. Sun has shown that systems implementing RG have identical WCRT to PM but benefit from a lower average response times as compared to other phase modification techniques.
The final analytical experiments examine the degree to which our Recurrence Model further reduces pessimism. This recurrence aware analysis attempts to reduce pessimism in WCRT estimation by eliminating some subtask preemptions by sibling subtasks that cannot logically occur6. The full description of this technique is beyond the scope of this paper. In our experiments we consider inter-subtask messages to be instantaneous. Analysis permits consideration of message transmission times and queuing and such delays can also easily be simulated. However, in a QoS enabled switched network environment network delays are expected to have very low variance, which we believe would lead to a limited impact on results obtained in this study. EXPERIMENTAL PLAN Determining the Best Analytical Bounds We begin our study by examining how subtask prioritization and subtask synchronization protocols affect the worst6
4
Synchron -ization Protocol
The scheduling techniques used in these experiments additionally include “shortest execution time” enhancements developed by [Liu] (but not included in the original synchronization protocol work performed by [Sun]) even though all subtasks are configured such that shortest execution time = longest execution times.
case response time in the hopes of finding a combination that provides the tightest bounds consistently. Impact of Subtask Prioritization on Algorithm Convergence — [4] has shown that the choice of subtask synchronization protocol affects the number of possible systems that can be analyzed. Specifically, the analysis algorithms for tasks employing direct synchronization do not always converge. However, the analysis algorithms for tasks employing phase modification are mathematically guaranteed to converge as long as the utilization rate is less than 100 percent on each processor. To make appropriate comparison between the techniques we must determine the subset of our 45,000 tasks for which direct synchronization algorithms converge. Fig. 1 shows the utilization rate versus convergence failures across all 45,000 tasks when employing direct synchronization and rate monotonic prioritization (DS_RM). Fig. 2 shows similar results across all 45,000 tasks when employing direct synchronization and proportional deadline monotonic prioritization (DS_PD) on the same scale. Specific data is given below. Switching to proportional deadline monotonic prioritization decreased divergence of the direct synchronization algorithm by two orders of magnitude. We expected the direct synchronization schedulability analysis (SA/DS) algorithm to converge more frequently with proportional deadline monotonic (PD) prioritization than with rate monotonic (RM) prioritization for several reasons. First, Sun's Experiment I, which compared prioritization methods using the phase modification schedulability analysis (SA/PM) algorithm, found that PD tends to yield smaller worst-case response time (WCRT) bounds than RM7. When applying SA/DS, we can understand divergence as obtaining infinite WCRT bounds. If PD and RM stand in the same relationship to one another with SA/DS as with SA/PM, then
Figure 2. Failure Rates for Tasks Employing Direct Synchronization and Proportional Deadline Monotonic Prioritization (DS_PD).
PD would tend to yield fewer infinite WCRT bounds, i.e., to converge more frequently. Second, as Sun described and illustrated with an example, SA/DS diverges when its outer loop behaves as a positive feedback loop. Each iteration calculates new bounds on intermediate end-to-end WCRTs (from an initial subtask through some not necessarily final subtask). When an intermediate WCRT increases from one iteration to the next, it provides more opportunities for preemption at the next stage of analysis, which may lead to another increase in the following iteration. In some cases positive feedback continues to increase intermediate WCRTs without limit, and SA/DS diverges. (In practice, we assume divergence after reaching some threshold.) Given that divergence of SA/DS is related to large intermediate WCRTs, and assuming that RM tends to yield larger WCRTs than PD with SA/PM and SA/DS, it is plausible that RM diverges more often than PD. Third, RM prioritization has an inherent characteristic that causes WCRT bounds to be overestimated. It assigns each subtask of a particular task the same priority. When the task has multiple subtasks assigned to the same processor (i.e., the task is recurrent on a processor), then RM does not establish a total priority ordering of subtasks on the processor. Lacking mechanisms for priority tie breaking, the schedulability analysis algorithms account for each of a pair of equal priority subtasks preempting the other. The introduction of these additional preemptions increases intermediate WCRTs, making it more likely for SA/DS to diverge. Convergence Rate for Tasks Employing Direct Synchronization and Rate Monotonic Prioritization (DS_RM)
Figure 1. Failure Rates for Tasks Employing Direct Synchronization and Rate Monotonic Prioritization (DS_RM).
7 Sun actually measured the worst-case schedulability index of a system of end-to-end tasks. This is the maximum over all tasks of the ratio of (the upper bound on) end-to-end response time to the task's period.
DS_RM: Total 45,000 Converged 35,472 Failed 9,528 DS_PD: Total 45,000 Converged 44,904 Failed 96
DS_RM experienced no failures below 55% processor utilization8 and DS_PD experienced no failures below 93% processor utilization. These results show that this analysis technique may be usable if processor loads can be restricted below a conservative threshold and if the pessimism in the predictions is tolerable. This is important because all commercial middleware technologies known to the authors strictly use direct synchronization. Impact of Subtask Prioritization on Predicted WCRT — [4] has shown that the choice of subtask prioritization affects the WCRT. We repeat that analysis for our 50 systems to establish initial upper bounds on the WCRT for our experimental tasks. First, we compare the performance of the direct synchronization algorithms under rate monotonic (DS_RM) and proportional deadline monotonic (DS_PD) for the 35472 tasks where both techniques converged. We compute a WCRT_index (WI) defined as WCRT/deadline. A WI greater that 1 indicates a task has missed its deadline or in other words is not scheduable9. Results for DS_RM and DS_PD across all 35,472 tasks are summarized below: Worst Case Response Time Index (WI) for DS_RM and DS_PD DS_RM: Min 0.031 Mean 0.532 Max 24.29 Var 0.701 Samples 35472 DS_PD: Min 0.048 Mean 0.549 Max 24.59 Var 0.437 Samples 44904 The maximum value of WI is nearly the same for both subtask prioritizations. However, the average value of WI is lower for DS_PD than for DS_RM indicating that DS_PD is typically better. To quantify the degree to which DS_PD is better, we examine the ratio of WI_DS_RM/WI_DS_PD for each of the 35472 tasks. When this ratio is greater than 1 it means that DS_RM predicting a more pessimistic WCRT than DS_PD. Per Task Ratio of Worst-Case Response Time Indexes (WI) for DS_RM Compared to DS_PD (WI_DS_RM/WI_DS_PD)
WI_DS_RM) pairs for all 35,472 tasks. The degree to which these points deviate from the diagonal represents the difference that these two priority assignment techniques make in the WCRT estimations. Next, we compare the performance of the phase modification algorithms under rate monotonic (PM_RM) and proportional deadline monotonic (PM_PD) for all 45000 tasks (PM algorithms always converge for processor utilizations less than 100%). Again, we compute a WCRT_index (WI) defined as WCRT/deadline. Results for PM_RM and PM_PD across all 45000 tasks are summarized below: Worst Case Response Time Index (WI) for PM_RM and PM_PD PM_RM: Min 0.031 Mean 0.406 Max 2.58 Var 0.048 Samples 45000 PM_PD: Min 0.048 Mean 0.383 Max 1.48 Var 0.036 Samples 45000 There is a general performance improvement from the proportional deadline monotonic prioritization. However, unlike in the direct synchronization case there is a substantial reduction in the maximum value as well. To quantify the degree to which PM_PD is better than PM_RM, we examine the ratio of WI_PM_RM/WI_PM_PD for each of the 45000 tasks. Per Task Ratio of Worst-Case Response Time Indexes (WI) for PM_RM Compared to PM_PD (WI_PM_RM/WI_PM_PD) Min 0.282 Mean 1.067 Max 7.054 Var 0.055 Samples 45000 30.951% of Samples < 1.0 (PM_RM better) 17.833% of Samples = 1.0 (no difference) 51.216% of Samples > 1.0 (PM_PD better) We note that PM_PD was as good or better than PM_RM in 69% of the tasks. This is a small improvement over the WI_DS_RM/WI_DS_PD ratio that was 63%. However, there is almost a 10x reduction in the maximum (the case where PD
Min 0.086 Mean 1.196 Max 67.92 Var 1.346 Samples 35472 36.181% of Samples < 1.0 (DS_RM better) 12.951% of Samples = 1.0 (no difference) 50.868% of Samples > 1.0 (DS_PD better) DS_PD was as good or better than DS_RM in more than 63% of the tasks. The maximum value indicates that in one instance DS_PD was 67.92 times better than DS_RM. The minimum value indicates that DS_RM was only 11.63 times better than DS_PD. Though DS_PD does better on average for a majority of cases, it is still not consistently better than DS_RM. Fig. 3 shows a "scatter plot" of the (WI_DS_PD, 8
In some cases the experimental data presented in this section has been rounded to three decimal places for formatting purposes. Those specific cases are denoted by italic emphasis. Unmodified values are available from the authors upon request. 9 The whole purpose of our study is to determine the degree of confidence designers should place in these scheduability predictions as they are known to be pessimistic.
Figure 3. Worst-Case Response Index (WI) for DS_PD Versus WI for DS_RM
offers the biggest improvement over RM) from 67.92 in the WI_DS_RM/WI_DS_PD ratio to 7.064 for WI_PM_RM/ WI_PM_PD. Similarly minimum (the case where RM offers the biggest improvement over PD) also reduced from 11.63 to 3.55. Again, proportional deadline monotonic prioritization offers improvements in a majority of cases. Fig. 4 shows a “scatter plot” of the (WI_PM_PD,WI_PM_RM) pairs for all 45000 tasks. In contrast to Fig. 3 we observe less scatter with most points falling within a well bounded region near the diagonal. Impact of Subtask Synchronization Protocol on Predicted WCRT — Thus far we have only compared rate monotonic and proportional deadline monotonic subtask prioritization for a particular synchronization protocol. The proportional deadline monotonic subtask prioritization yields as good or better results on average in the majority of cases. So for proportional deadline monotonic subtask prioritization we consider the relative performance between the direct synchronization protocol (DS_PD) and the phase modification protocol (PM_PD). The ratio of WCRT Indexes (WI) for the 44904 cases for which both DS_PD and PM_PD converged is summarized below. Per Task Ratio of Worst Case Response Time Indexes (WI) for DS_PD Compared to PM_PD (WI_DS_PD/WI_PM_PD) Min 1.0 Mean 1.302 Max 43.34 Var 0.650 Samples 35472 13084: 29.1% of Samples = 1.0 (no difference) 31820: 70.9% of Samples > 1.0 (PM_PD better) The minimum is 1.0, which means there was no case where DS_PD performed better than PM_PD. We also note that the maximum is 43.34, which means that there was at least one case where DS_PD was 43 times more pessimistic than PM_PD. We see some improvement in 70.9% of the cases. Fig. 5 shows a “scatter plot” of the of the (WI_PM_PD, WI_DS_PD) pairs for the 44,904 tasks. For this figure we
Figure 5. Worst-Case Response Index (WI) for PM_PD Versus WI for DS_PD
switch to a semi log format as DS_PD values have a much larger range than PM_PD. We draw several conclusions from these experiments. One, proportional deadline monotonic subtask prioritization produces less pessimistic WCRT in the majority of cases. And when rate monotonic subtask prioritization is less pessimistic, it is less of an improvement. Two, phase modification synchronization protocols produce far less pessimistic WCRT in the majority of cases and are never more pessimistic than direct synchronization. However, the question still remains how pessimistic are they? In the next section we will present simulation results that will allows us to quantify the pessimism for DS_PD and PM_PD10. QUANTIFYING PESSIMISM Having experimented with analysis techniques we have two upper bounds for WCRT (DS_PD and PM_PD). To quantify pessimism of these analytical bounds we compare them with corresponding lower bounds on WCRT. We obtain these lower bounds from discrete event simulations. Our discrete event simulations select a random phase for the initial release of each periodic task in a system. We execute the simulation of each system for 20 different, randomly selected initial phase values. Since periods are randomly generated, non-integer values we have noted that it is not necessary to run many different initial phases to achieve stable values for WCRT (In the simulation WCRT is not a predicted value as in the analysis. Here WCRT is recorded as the longest task completion (or end-to-end response) time observed). As the simulation executes, new instances of tasks are released at each respective period. The simulation implements preemptive scheduling at each processor. We also track how many periods it has been since a task's WCRT last increased. The simulation runs until the WCRT values recorded do not increase for any tasks for at least 1,000 periods. (Since periods
Figure 4. Worst-Case Response Index (WI) for PM_PD Versus WI for PM_RM
10
We excluded DS_RM and PM_RM because the *_PD bounds are tighter in the majority of cases.
are different when we reach this condition all but one task will have observed unchanging WCRT for (much) more than 1000 periods.) Generally this condition of stable WCRT values arises only after hundreds of thousands of periods for a given initial phase. We have no guarantees that the simulation will ever experience the true WCRT. This is acceptable because we are only interested in obtaining a lower bound. If the upper (analysis) and lower (simulation) bounds are very close to each other we have tightly bounded the true WCRT. However, if the gap between upper and lower bound is large we have greater uncertainty in the WCRT but it still has quantified the maximum pessimism. Just as we had previously examined WCRT Index (WI) (WI = WCRT/Deadline) statistics for the analysis experiments we now examine WI statistics for the simulations. We append "_s" to the experiment name to designate simulation and we include the corresponding analytical results for each of comparisons: Worst Case Response Time Index (WI) for DS_PD_s and RG_PD_s DS_PD_s: Min 0.046 Mean 0.380 Max 1.606 Var 0.039 Samples 45000 DS_PD : Min 0.048 Mean 0.446 Max 24.59 Var 0.231 Samples 4490411 RG_PD_s: Min 0.046 Mean 0.375 Max 1.478 Var 0.036 Samples 45000 PM_PD : Min 0.048 Mean 0.383 Max 1.480 Var 0.036 Samples 45000 Considering DS_PD first, we note that the simulation minimum and mean WI are close to the analysis values but there is a huge discrepancy in the maximum values. This means that there is at least one case where the quantifying the pessimism was virtually worthless as the gap between 24.59 and 1.60612 is almost as large as the original gap between 24.59 and 0. Now considering PM_PD, we not only see agreements on the minimum and mean values but we also see very close agreement between the maximum values. In the worst case, the PM_PD analytical bound is much tighter (50% deadline overrun) than the DS_PD bound (2400% deadline overrun). Fig. 6 show the per task comparison of analytical and simulated WI for both DS_PD and PM_PD. The left side shows a scatter plot of (WI_DS_PD_s, WI_DS_PD) ordered pairs and the right side shows a scatter plot of (WI_RG_PD_s, WI_PM_PD) ordered pairs. This figures graphically depicts the gap or quantified pessimism of each techniques across all 44906 and 45000 tasks respectively. When we show two such scatter plots side by side we also obtain a sense of relative tightness of the 2 bounds (note that y-axis is logscale). 11
DS_PD analysis converged for 44904 but all the tasks execute in the simulation. 12 It is important to note that from these particular statistics we cannot assume the task that generated the maximum value in the simulation is the task that generated the maximum value in the analysis. However, we use the maximum value of the simulation 1.606 because the maximum gap is at least that large.
Figure 6. Quantified Pessimism in DS_PD (left side) and Quantified Pessimism in PM_PD (right side)
In this figure we see clearly that DS_PD technique exhibits a very large gap between its upper bound and lower bound while PM_PD provides a tight range between its upper and lower bound. The situation is worse than what Fig. 6 shows because there were 96 cases which did not converge for DS_PD analysis. RECURRENCE AWARE ANALYSIS IMPROVEMENTS In the Experimental Methodology Section we briefly discussed how subtask recurrence leads to pessimistic analysis estimates of WCRT. In this section we quantify how our improved analysis techniques that consider subtask recurrence can lower WCRT estimates. First, we define average recurrence degree (just recurrence for short) of a system of tasks using the following algorithm: sum = 0; foreach task { foreach processor_which_hosts_subtasks_of_this_task { nt = number_of_subtasks_of_this_task_on_this_processor sum += nt*nt } } recurrence = sum/total__number_of_subtasks_for_all_tasks With the above definition if a task with four subtasks is distributed such that two subtasks end up on each of two processors the recurrence degree will be two. If all four subtasks are on one processor the recurrence degree will be four. If all subtasks are on their own processor the recurrence degree will be one. Exploring the Benefits of Recurrence Aware Analysis Techniques — Before applying the recurrence based analysis techniques to our 50 systems we conducted a smaller experiment to quantify the range of improvements we should expect. In this experiment we held constant the number of tasks (NT) at eight and the processor loading at 50%. We varied the number of processors (NP) from one to four and the
number of subtasks (ST) per task from one to 16. Subtask assignments to processor(s) were random. For each value of (NP, ST) combination we generated 1000 different systems13 and examined the WI_PM_PD/WI_R_PM_PD statistic. Fig. 7 shows the average value for this ratio as a function of average recurrence degree over the 1000 systems being analyzed. For a given value of NP, recurrence degree increases as number of subtasks increase. The maximum improvement is about 33% (for NP=1) while for other cases improvement peaks at ~15%20%. Fig. 7 further shows the maximum improvement decreases as a function of number of processors. Fig. 7 also shows that above a certain recurrence degree the improvement declines. A possible explanation for this might be that the processor loading is held constant at 50%, which causes each subtask execution to get smaller as the total number of subtasks increase. Smaller subtask execution times may reduce opportunities for subtask preemption even when a large number of tasks share a single processor. However, the root cause of this behavior is not understood at this time. While the modest improvement from the recurrent aware analysis algorithm is welcome, we must also understand the cost of this approach. We collected run times for both the PM_PD and R_PM_PD analysis algorithms for all the cases that are summarized in Fig. 7. These algorithm execution times are shown in Fig. 8. We note that compared to PM_PD the R_PM_PD has a considerably larger execution time. In general the execution time increases as the recurrence degree increases while for base PM algorithm the run time is fairly constant. However, the size of the relative difference in R_PM_PD and PM_PD run times may not be a concern. The maximum execution time observed for R_PM_PD was 2.5 seconds. This occurred for a 128-subtask system. This indicates that R_PM_ PD will be a practical technique for many dynamically as well as statically scheduled systems. Furthermore, we have yet to attempt to optimize our implementation of R_PM_PD.
Figure 8. Analysis Algorithm CPU Usage Comparison.
Quatifying Benefits of Recurrence Aware Techniques — We now seek to measure the benefit that the recurrence aware analysis algorithm provides for our original 50 systems each consisting of 24 processors and 48 total subtasks. Since the number of processors is relatively large compared to the number of subtasks and subtasks are assigned randomly to processors, we expect relatively low degrees of recurrence. Recurrence Degree Statistics Across All 50 Systems Min 1 Mean 1.133 Max 1.417 Var 0.00593 Samples 50 The average recurrence is 1.13, which is low. In general, for systems with these characteristics we would expect only modest improvement by using the recurrence aware version of the algorithm. Below we summarize WI statistics for R_PM_PD and for comparison we include PM_PD. Worst Case Response Time Index (WI) for R_PM_PD and PM_PD R_PM_PD: Min 0.046 Mean 0.381 Max 1.478 Var 0.036 Samples 45000 PM_PD: Min 0.048 Mean 0.383 Max 1.478 Var 0.036 Samples 45000 Again we compute the ratio WI_PM_PD/WI_R_PM_PD for each task. These are summarized below: Per Task Ratio of Worst Case Response Time Indexes (WI) for PM_PD Compared to R_PM_PD (WI_PM_PD/WI_R_PM_PD) Min 1.0 Mean 1.013 Max 1.463 Var 0.003 Samples 45000 40758: 90.6% of Samples = 1.0 (no difference) 4242: 9.4% of Samples > 1.0 (R_PM_PD better)
Figure 7. Average Improvement in Upper bound WCRT Estimates by Exploiting Knowledge of Recurrence.
13 We are able to deal with such a large number since we do not do any simulations of these systems. Running simulations is much more time consuming than analysis.
Overall the WI statistics are little changed when we move to R_PM_PD from PM_PD. The ratio of WI_R_PM_PD/WI_ PM_PD reveals that average improvement is only 1.3% though the maximum improvement was 46.3%, which is valuable. We now present a final graph, Fig. 9, depicting the cumulative distribution of the ratio WI_R_PM_PD/WI_ RG_PD_s which quantifies the pessimism for the best analysis
TABLE 4. SUMMARY WI STATISTICS FOR INDIVIDUAL EXPERIMENTS Experiment DS_RM DS_PD PM_RM PM_PD R_PM_RM R_PM_PD DS_RM_sa DS_PD_sb RG_RM_sb RG_PD_sb
Minimum 0.03088 0.04761 0.03088 0.04761 0.03088 0.04563 0.03088 0.04563 0.03088 0.04563
Mean 0.5322 0.5488 0.406 0.3829 0.3994 0.3807 0.3873 0.3802 0.3806 0.3755
Maximum 24.29 24.59 2.58 1.478 2.58 1.478 6.071 1.606 2.493 1.478
Variance 0.7013 0.4374 0.04771 0.03586 0.04998 0.03666 0.0534 0.03917 0.04293 0.03551
Samples 35472 44904 45000 45000 45000 45000 45000 45000 45000 45000
a. Simulation results were generated using 20 different initial phases for each task for all 50 systems. b. We are able to deal with such a large number since we do not do any simulations of these systems. Running simulations is much more time consuming than analysis.
Figure 9. Cumulative Distribution of WI_R_PM_PD/WI_RG_PD_s.
technique that is available to us. The average of this ratio is only 1.013, and the maximum is 1.463, but an examination of the cumulative distribution provides more insight. We note that for about 90% of cases this ratio is less than 1.03, in 91% the ratio is less than 1.05, 94.5% are less than 1.1, and 99% are less than 1.2. From this we conclude that pessimism is reasonably bounded for most cases. Finally, we with aid of Fig. 10 we examine how pessimism relates to processor utilization. We note that while pessimism does generally increase with processor utilization this is a weak relationship with ratio of upper bound to lower bound reaching 1.3 at processor utilizations around 50%. SUMMARY TABLES We provide tables that summarize the worst-case response index (WI) statistics for all our experiments (Table 4) and the per task WI ratios for each meaningful pair-wise combination of experiments (Table 5). In Table 5 each row lists two experi-
ments and the statistics for the WI_exp1/WI_exp2 ratios. The last column of Table 5 contains three sets of numbers where each set consists of a count and a percentage. The three sets represent the number of cases where the “exp_1” WI value was smaller than, equal to, or greater than the corresponding value for “exp_2.” The three sets are separated by a “/” character. These tables an invaluable reference in our discus-sions and in authoring this paper. CONCLUSIONS We have quantified the pessimism in distributed scheduling analysis algorithms by examining the difference between upper and lower bound estimates on WCRT. When the differences between upper and lower bounds are small (e.g. < a factor of 2) the system integrator can have confidence that the predicted worst-case response times (WCRT) are not pessimistic. Our experiments have shown that non-greedy synchronization techniques greatly reduce pessimism compared to direct synchronization techniques. Non-greedy techniques can achieve very tight bounds in almost all cases. However, compared to direct synchronization, phase modification techniques require changes to the dispatching mechanisms. Ideally these changes can be incorporated at the appropriate layer of middleware (otherwise the applications themselves must deal with this issue). Alternatively, direct synchronization may yield an acceptable gap between upper and lower bounds for low utilization, though this needs further study. In the interim a practical approach is to analyze all systems using both techniques and then selecting the best approach based on the WCRT for critical tasks. We believe that using model based or model integrated techniques a system integrator could easily annotate system models and conduct these analyses. This ability will permit systems to be deployed—and changed—with much less time and expense for testing to “validate” that end-to-end response times will be met.
Figure 10. Pessimism as a Function of Processor Utilization.
A basic prerequisite of this approach is the need to tightly characterize the subtask execution times and task periods. Systems that cannot will be less able to provide nonpessimistic assurances (if any) about end-to-end response times. One limitation of this technique is that it does not apply to aperiodic tasks.
TABLE 5 WI RATIOS (WIEXP1/WIEXP2) FOR PAIR-WISE COMBINATION OF EXPERIMENTS Exp1
Exp2
Min.
Mean
Max.
Variance Samples
DS_RM
DS_PD
0.086
1.196
67.92
1.346
35472
DS_RM
PM_RM
1.0
1.309
42.03
1.085
35472
DS_RM
DS_RM_s
1.0
1.437
66.37
1.741
35472
DS_PD
PM_PD
1.0
1.302
43.34
0.6497
44904
DS_PD
DS_PD_s
1.0
1.327
42.28
0.6562
44904
PM_RM
PM_PD
0.282
1.067
7.054
0.0552
45000
PM_RM
R_PM_RM
1.0
1.039
2.142
0.01883
45000
PM_RM
RG_RM_s
1.0
1.083
2.731
0.03159
45000
PM_PD
R_PM_PD
1.0
1.013
1.463
0.002583
45000
PM_PD
RG_PD_s
1.0
1.026
1.463
0.004023
45000
R_PM_RM
RG_RM_s
1.0
1.044
2.731
0.01602
45000
R_PM_PD
RG_PD_s
1.0
1.013
1.463
0.001756
45000
DS_RM_s
DS_PD_s
0.2412
1.021
17.01
0.07469
45000
DS_RM_s
RG_RM_s
0.6152
1.005
3.453
0.006473
45000
DS_PD_s
RG_PD_s
0.6899
1.005
1.825
0.00274
45000
RG_RM_s
RG_PD_s
0.2276
1.018
7.054
0.04645
45000
Our experience working on next generation combat systems has shown many cases where periodic and sporadic tasks can be characterized in sufficient detail to use these techniques. In many of those cases, guarantees on WCRT are required to certify the system fit for combat. Certification is accomplished today by heuristic analysis and exhaustive testing (the Aegis Combat System undergoes two years of acceptance testing prior to deployment). We believe these nonpessimistic techniques can eliminate the heuristic analysis and greatly reduce testing times. FUTURE WORK There are a number of directions this work can be extended, and we are working on some of these or have near terms plans to do so.
Below/Equal/Above [count percent] 12834 36.2% 4594 13% 18044 50.7% 0 0% 12207 34.413% 23265 65.5869% 0 0% 8926 25.1635% 26546 74.8364% 0 0% 13084 29.1377% 31820 70.8622% 0 0% 10057 22.3966% 34847 77.6033% 13928 30.9511% 8025 17.8333% 23047 51.2155% 0 0% 40194 89.32% 4806 10.68% 0 0% 32829 72.9533% 12171 27.0466% 0 0% 40758 90.5733% 4242 9.4266% 0 0% 35192 78.2044% 9808 21.7955% 0 0% 37386 83.08% 7614 16.92% 0 0% 39301 87.3355% 5699 12.6644% 18557 41.2377% 8111 18.0244% 18332 40.7377% 9483 21.0733% 29003 64.4511% 6514 14.4755% 7997 17.7711% 28454 63.2311% 8549 18.9977% 16440 36.5333% 10036 22.3022% 18524 41.1644%
We seek to find ways to combine this periodic model with aperiodic task models from Abdelzaher and Lu to provide an integrated analytical solution that remains practical. We want to extend these periodic techniques to handle some degree of dynamic variation. This includes changes in resource requirements (e.g. jitter in subtask execution times or period) and resource availability (e.g. a partial failure of compute infrastructure.) While we did not present results in this paper, the current techniques handle subtask execution times that vary over a range of values. However, they still lack the ability to produce a probability distribution for worst-case response times (WCRT) in the cases where maximum subtask execution times cannot be practically specified. If possible, such techniques would permit the system integrator to tradeoff
risk versus cost and would provide an excellent tool for quantifying soft real-time performance. To quantify how well these analytical results hold for real systems, we plan to develop a release guard enabled CIAO container and experimentally evaluate the software in a setting such as Emulab. There is also a need to integrate this approach with system development life cycle, which includes requirements capture, analysis, design, development, testing, integration, and deployment. We have a modest near term goal of integrating our work in an environment such as Cadena and an ongoing effort to influence OMG and other standards bodies in these area. The system model can be further extended to consider network delays and also to consider task chains that contain forks and joins. We believe while these add complexity it seems possible to extend analysis to handle these cases.
Finally, we plan to formally document recurrence aware analysis techniques in a separate paper. REFERENCES [1] Tarek Abdelzaher and Chenyang Lu, “Schedulability Analysis and Utilization Bounds for Highly Scalable Real-Time Services,'' IEEE RealTime Technology and Applications Symposium, TaiPei, Taiwan, June 2001. [2] G. Thaker, and P. Lardieri, “In Search of Hard Real-Time COTS Distributed Object Computing Middleware,” Distributed Objects and Applications Conference, Rome, Italy, 2001. [3] G. Thaker, P. Lardieri, D. Krecker, K. O’Hara, and C. Winters, “Achieving Bounded End-to-End Latencies with Real-time Linux and Real-time CORBA,” Second Annual OMG Workshop on Real-Time and Embedded Systems, Arlington, VA, September 2002. [4] J. Sun, “Fixed-Priority End-to-End Scheduling in Distributed Real-Time Systems,” Ph.D. thesis, Department of Computer Science, University of Illinois at Urbana-Champagne, 1997.
