Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Alternatives for Scheduling Virtual Machines in Real-Time Embedded Systems Robert Kaiser Distributed Systems Lab University of Applied Sciences Wiesbaden, Germany
IIES 08, Glasgow, 1-Apr-2008
Robert Kaiser
[email protected]
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Outline
1
Introduction
2
Timing properties of virtual machines Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
3
Supporting real-time systems Workload classification Requirements of different load classes Scheduling infrastructure
4
Conclusion
Robert Kaiser
[email protected]
1 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Outline
1
Introduction
2
Timing properties of virtual machines Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
3
Supporting real-time systems Workload classification Requirements of different load classes Scheduling infrastructure
4
Conclusion
Robert Kaiser
[email protected]
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Motivation(1)
Embedded systems are becoming more powerful Disproprotionate relation of performance vs. cost (e.g. 2 x cost ⇒ 10 x performance) Using this potential makes sense .. .. economically: same/more performance at lower cost .. technically: fewer components ⇒ fewer points of failure
⇒ More functionality per controller Complexity must be managed, though...
Robert Kaiser
[email protected]
2 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Motivation(2)
Software of embedded systems becomes more complex Applications from different suppliers .. .. may serve different (unrelated) purposes .. may not be aware of each other .. need to be securely isolated (liability, safety, security)
⇒ A case for virtualisation!
Robert Kaiser
[email protected]
3 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
A Case for Virtualisation Complex embedded systems need management Applications must be kept from interfering with each other
→ Need fault containment Applications may be programmed against different OS APIs
→ Need to support multiple OSs in parallel
⇒ Similar requirements as server consolidation Many embedded systems already have the necessary resources (e.g. MMU/MPU) But can virtualisation be applied to embedded systems ”as is”? Workload is often known in advance
→ Dynamic resource management is less important Many applications have soft/hard timing requirements
⇒ Need to support real-time guest systems! Robert Kaiser
[email protected]
4 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Outline
1
Introduction
2
Timing properties of virtual machines Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
3
Supporting real-time systems Workload classification Requirements of different load classes Scheduling infrastructure
4
Conclusion
Robert Kaiser
[email protected]
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Hierarchical scheduler
Virtual machine monitor acts as a scheduler Virtual machines host guest OSs which –again– act as schedulers
→ VMM is the first level in a scheduler hierarchy VMM is unaware of this: ”VMs are just processes” Goal: share the CPU among VMs
Robert Kaiser
60 % CPU
30 % CPU
10 % CPU
Guest OS
Guest OS
Guest OS
Local Scheduler
Local Scheduler
Processes
Local Scheduler
VM−Scheduler
VMM/Hypervisor Hardware
[email protected]
5 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM 2
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
CPU share 66% 33%
VM 1 VM 2 real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
2
active VM 2 1
real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
2
active VM 2 1
real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
2
active VM 2 1
real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
2
active VM 2 1
real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Proportional share scheduling
Ideal: VMs continuously share CPU
VM exec time
VM 1
Reality: approximation by time multiplexing VM
Smaller quanta: → better approximation ... but also: more overhead - Scheduler invocations - Cache & TLB losses
2
active VM 2 1
real world time
Robert Kaiser
[email protected]
6 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Effect on latency Comparing RT process running on real hardware vs. virtual machine Virtual machine may experience a ”blackout” Worst case delay: Tdel = (N − 1) · Tvm + N · Tsw T
000sw 111 111 000 Case I: no Virtualisation
1111111111111111111111111111 0000000000000000000000000000
r j sj lj
Case II: Proc. in VM0 (worst case)
1111111 0000000
rj
sj pvm T
11 Tvm000 00 111sw 00 11Tvm VM0
VM1
T
T
00sw000 11 111 111sw 00 000 11 Tvm VM2
Robert Kaiser
T
00sw00 11 11 Tvm 000 111 VM0
[email protected]
7 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Effect on latency Comparing RT process running on real hardware vs. virtual machine Virtual machine may experience a ”blackout” Worst case delay: Tdel = (N − 1) · Tvm + N · Tsw T
000sw 111 111 000 Case I: no Virtualisation
1111111111111111111111111111 0000000000000000000000000000
r j sj lj
Case II: Proc. in VM0 (worst case)
1111111 0000000
BLACKOUT!
rj
sj pvm T
11 Tvm000 00 111sw 00 11Tvm VM0
VM1
T
T
00sw000 11 111 111sw 00 000 11 Tvm VM2
Robert Kaiser
T
00sw00 11 11 Tvm 000 111 VM0
[email protected]
7 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Effect on latency Comparing RT process running on real hardware vs. virtual machine Virtual machine may experience a ”blackout” Worst case delay: Tdel = (N − 1) · Tvm + N · Tsw T
000sw 111 111 000 Case I: no Virtualisation
1111111111111111111111111111 0000000000000000000000000000
r j sj lj
Case II: Proc. in VM0 (worst case)
1111111 0000000
BLACKOUT!
rj
sj pvm T
11 Tvm000 00 111sw 00 11Tvm VM0
VM1
T
T
00sw000 11 111 111sw 00 000 11 Tvm VM2
Robert Kaiser
T
00sw00 11 11 Tvm 000 111 VM0
[email protected]
7 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
Simulated overheads as function of quantum Simulateda Scheduler: VTRR (proportional share) fflood favg delay
Overhead [%]
80 60 40 20 0
10
100 1000 Quantum [µs]
55 50 45 40 35 30 25 20 15 10 5 0 10000
⇒ Observation: for quanta ≤ 1 ms: Delay bound [ms]
100
(3 VMs, CPU:
[email protected], 64k WSS)
overhead becomes significant Delay: proportional to quantum and number of VMs Here: for quanta ≥ 1 ms
⇒ Delay ≥ 5 ms Contemporary RTOS: ∼5-10µ s
a See: Kaiser: Empirische Ermittlung Cache-bedingter Umschaltverluste
Robert Kaiser
[email protected]
8 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Outline
1
Introduction
2
Timing properties of virtual machines Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
3
Supporting real-time systems Workload classification Requirements of different load classes Scheduling infrastructure
4
Conclusion
Robert Kaiser
[email protected]
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Upshot
Using proportional share scheduling, the delay/jitter imposed by virtualisation is severe, but bounded. For many practical real-time applications, a ten to hundred millisecond delay or jitter is entirely acceptable.
⇒ For these, virtualisation is applicable ”as is” But: there also exist applications for which such delay/jitter is unacceptable. How to support these, too? How to integrate this variety of requirements?
Robert Kaiser
[email protected]
9 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Mixture of workloads to support Real-time: must1 or should2 meet deadlines Two subclasses: Time-driven: static schedule, typically periodic Event-driven: scheduled in response to (unpredictable) events, assumed to be sporadic
Non real-time: no need to meet deadlines, instead, try to utilise all resources Assumption: Each class uses a specific OS API
⇒ Guests and their VMs as a whole can be classified as one of: 1 2 3
1 2
Time-driven, real-time Event-driven, real-time Non real-time
= ”hard’ real-time = ”soft” real-time Robert Kaiser
[email protected]
10 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Supporting non real-time VMs
No deadlines to meet → VM delay not a problem If multiple non-RT VMs exist: must share CPU evenly Should utilise all available resources → self-suspend when not ready (→ VMM schedule becomes unpredictable) Allocation of time to real-time VMs is designed for the worst case Worst case occurs rarely (if at all) In the average case, real-time VMs will not use all of their allocated time
⇒ Dynamically re-assign unused time to non real-time VMs Real-time VMs take priority over non real-time VMs May need provisions to avoid starvation, though..
Robert Kaiser
[email protected]
11 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Supporting time-driven VMs Cause of VM delay: VMM schedule and local schedules not correlated
⇒ Synchronise VMM schedule and local schedules of time-driven VMs Define VMM schedule to ”enclose” all time-driven local schedules Restrictions: Local schedules must not overlap Local schedules must use same (or harmonic) periods
σ1(t) 3 2 1
σ2(t)
t
3 2 1
σ3(t)
t
3 2 1
σvm(t)
t
3 2 1
Low jitter (e.g. for PLCs)
t
Resulting ”super schedule” is strictly a function of time Robert Kaiser
[email protected]
12 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Supporting event-driven VMs
Event-driven VMs need access to CPU at arbitrary times
⇒ Need ability to preempt current VM Conflicts with time-driven VMs Two choices: Give event-driven VMs precedence over time-driven VMs → Time-driven VMs experience jitter and delays Give time-driven VMs precedence over event-driven VMs → Event-driven VMs are delayed
Classical dilemma: no generic solution (for uniprocessor architectures) Approach must be flexible enough to allow both choices on a case by case basis
Robert Kaiser
[email protected]
13 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Combining time-driven, priority-based and prop. share
Assign static priorities to VMs
Ready queues prio
Assign VMs to time domains (τi ) (represented by a set of ready queues, one per priority)
.............
..... τ0 ..... 0
prio
..... .............
τ1 .....
A VM is scheduled iff
1 2
.....
.................
- its time domain is active, and - it has the highest priority
prio
.............
..... τ N−1
prio>?
dispatch
N−2 N−1
σvm(t)
.....
Up to two time domains can be active at a time: τ0 : background domain, always active τi , i = 0..N: foreground domain, cyclically switched VMs from foreground and background domain compete by priority VMs of same domain and priority are scheduled to proportional share
Robert Kaiser
[email protected]
14 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Semantics of priorities and time domain time-driven VMs: Foreground domain, individual, high priorities ⇒ individual, time-driven schedules
prio
event-driven VMs:
2
Background domain, individual, high priorities ⇒ can request CPU any time (iff sufficient priority)
non real-time VMs:
1
σ vm (t)
3
t
3 2 1
Background domain, common, low priority ⇒ share all unused CPU resources
Robert Kaiser
[email protected]
15 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Semantics of priorities and time domain time-driven VMs: Foreground domain, individual, high priorities ⇒ individual, time-driven schedules
prio
4
event-driven VMs:
2
Background domain, individual, high priorities ⇒ can request CPU any time (iff sufficient priority)
non real-time VMs:
5 1
σ vm (t)
3
t
3 2 1
Background domain, common, low priority ⇒ share all unused CPU resources
Robert Kaiser
[email protected]
15 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Semantics of priorities and time domain time-driven VMs: Foreground domain, individual, high priorities ⇒ individual, time-driven schedules
prio
4
event-driven VMs:
2
Background domain, individual, high priorities ⇒ can request CPU any time (iff sufficient priority)
non real-time VMs:
5 1 6 σ vm (t)
3 7 t
3 2 1
Background domain, common, low priority ⇒ share all unused CPU resources
Robert Kaiser
[email protected]
15 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Workload classification Requirements of different load classes Scheduling infrastructure
Implementation details
First implementation: ”PikeOS” µ Kernel (Sysgo AG) (Round robin instead of proportional share)
Ready queues prio
Use cases so far: RTOS & Linux, PLC & Linux Priorities control (individually) precedence of time-driven vs. event-driven VMs
.............
..... τ0 ..... 0
prio
..... .............
τ1 .....
1 2
.....
................. prio
.............
..... τ N−1
prio>?
dispatch
N−2 N−1
σvm(t)
.....
→ O(1) complexity High priority VMs trusted not to exceed their budgets
Robert Kaiser
[email protected]
16 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Outline
1
Introduction
2
Timing properties of virtual machines Hierarchical scheduling Proportional share scheduling Effect on latency Overhead vs. Latency
3
Supporting real-time systems Workload classification Requirements of different load classes Scheduling infrastructure
4
Conclusion
Robert Kaiser
[email protected]
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Summary
Applying virtualisation to embedded systems makes sense Current virtualisation approaches have limited real-time support Proposed approach can improve it (with some restrictions) Coexistence of time-driven and event-driven subsystems remains problematic Proposed approach can be implemented efficiently by means of the outlined scheduler infrastructure
Robert Kaiser
[email protected]
17 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
Further work
Use Xen as testbed Introduce budgets to limit high priority time consumption Multiprocessor (MultiCore) support MultiCore CPUs are attractive for embedded systems Decouple time-driven and event-driven VMs Enable parallel programming (”coscheduling”) under guest OS control
Apply schedulability analysis to guest OSs RMS: already covered (though not in this paper) EDF: TBD
Improve estimates for cache-related switch overheads Validate simulation-based estimation under realistic workloads Estimates should also be applicable processor migration cost
Robert Kaiser
[email protected]
18 / 18
Introduction Timing properties of virtual machines Supporting real-time systems Conclusion
The End
Thank you for your attention!
Any Questions? Robert Kaiser
[email protected]
18 / 18
![Presentation Slides - Sony Global [PDF]](https://m.moam.info/img/260x300/presentation-slides-sony-global-pdf_648c182e098a9ea83a8b45bc.jpg)
