AIAA-2000-4593
REAL-TIME SIMULATION, CONTROL AND HIL WITH COTS COMPUTING CLUSTERS Marco Papini, M.A.Sc., Aeronautics and Paul Baracos, Ph.D. P.Eng. Opal-RT, 1751 Richardson, suite 2525, Montreal, QC, Canada H3K 1G6 phone:514-935-2323 email:
[email protected] http://www.Opal-RT.com
Abstract This paper describes technology for high-performance simulation. Challenging applications such as real-time hardware-in-the-loop and mega-simulation are made possible using parallel architectures based on COTS PC components. The system described is flexible enough to allow expansion and reconfiguration, and covers the entire design process from modeling to real-time simulation or control without the user ever having to write code. Keywords Real-time simulation, hardware-in-the-loop simulation, distributed simulation megasimulation, simulation accelerator, computing cluster, COTS
RT[3] and the Defense Research Establishment[4] have endeavored for the last three years to create technology to simplify this process. The technology enables model decomposition to allow parallel execution and automatically generates, downloads and runs code that meets above-mentioned constraints, without forcing the user to write any target code.
Acronyms and Abbreviations The following acronyms and abbreviation are used in this paper: ARINC COTS HIL PC RT-Lab TCP/IP UDP
Overview Aeronautics has long been at the forefront of simulation and control, employing the latest in (and most expensive) computing technologies. Early analog computers gave way to digital technology, requiring in-house software departments to assemble custom software tools for their engineers to apply to ever more complex problems. Today, COTS software and hardware allow large savings in cost and engineering effort. Commercial simulation packages such as Matlab/Simulink™ 1 and MATRIXX/SystemBuild™ 2 are now widely used in the aeronautics industry. They have become the modeling tools of choice because their graphic interface mimics the function-block methodologies taught in universities and technical school. This reduces the learning time and increases engineering productivity. Another useful feature of these packages is automatic code generation. This frees engineers from the tedious and error-prone task of writing code. They make it possible to go from concept to simulation without ever having to write code. However, the code generated by these packages is not always well suited to real-time execution, HIL applications or parallel processing. Some form of hand coding is still required.
Aeronautical Radio Inc., a nonprofit avionics standardization organization Commercial-off-the-shelf Hardware-In-the-Loop simulation Personal Computer (IBM compatible) Opal-RT simulation toolset Transmission Control Protocol/ Internet Protocol User Datagram Protocol
Introduction Real-time simulation and support for HIL modeling are increasingly recognized as essential tools for aeronautic design. Pollini and Innocenti[1] review the state of the art and point out the necessity of distributed simulation for dealing with complex dynamic models and real-time constraints. They discuss topics such as simulator decomposition, computational needs, inter-processor bandwidth, TCP/IP vs. UDP, deadlock avoidance, real-time clock synchronization and optimization, and console integration. Stasko et al.[2] acknowledge that HIL is no longer a luxury in modern system design. They discuss its many benefits and methods of implementation. Simulation software is notoriously difficult to write, debug and maintain. Real-time, high-performance, deadlock-free distributed code is even harder. Opal-
™ ™
1 2
Trademark of The MathWorks Trademark of Wind River Systems
Copyright © 2000 by Opal-RT Technologies Inc. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.
1 American Institute of Aeronautics and Astronautics
As COTS software has replaced custom development, the PC has now supplanted workstations and even mainframes for many simulation and control applications. With the advent of PC clusters, embedded PCs, high-end graphics cards and other technologies, the range of applications that can benefit from the low cost of COTS hardware has grown.
However, off-line simulation does not validate the implementation of the controller. In a sampled-time digital computer with real-life I/O there is no guarantee that the real-time performance constraints can be met.
desired trajectory
Guiding Principles The research and development described in this paper has been guided by the following principles: • Advanced modeling and simulation algorithms (not hardware) will be the key to success. • Real-time simulation and control should be easy and affordable. • The tools should look after low-level details, taking the user from graphical models to hardware without hand coding. • Benefit from the market-driven advances in COTS components. Use the latest technologies as soon as they appear. • Simulation architecture should be scalable. • Models should be deployable in real-life as the actual control system. • In real-time, every microsecond counts.
virtual controller virtual aircraft
IBM Compatible
Figure 2. Virtual aircraft system (simulated)
Hardware-in-the-loop In order to validate that a controller can indeed meet real real-time performance constraints, the HIL technique is employed. In HIL, either a simulated plant is wired to a real controller, Figure 3, or a real plant to a simulated controller, Figure 4. In either case, the distinguishing feature is the presence of I/O and wires.
Design by Simulation Aircraft systems typically contain numerous feedback loops, each consisting of a physical system or plant to be controlled, such as sensors, actuators, a controller and a setpoint. The controller is in fact a computer that uses input data from the sensors to calculate an output control signal that moves the actuators in such a way as to the follow the desired state trajectory, as shown in Figure 1.
simulated trajectory
virtual aircraft
input signals from simulator
real I/O controller
I/O control signals to simulator
IBM Compatible
desired trajectory
Figure 3. HIL: simulated plant, real controller
input signals
controller I/O control signals
×
sensors +
actuators
real trajectory + +
-
+
d dt
simulated controller
aircraft
Figure 1. Aircraft control loop (real, not simulated)
input signals from plant
I/O IBM Compatible
Due to the complexity of the dynamic equations, it is rarely possible to find closed-form solutions. The alternative is to carry out simulation studies, entirely within the virtual world of a desktop computer, as shown in Figure 2.
real aircraft
control signals to plant
Figure 4. HIL: real plant, simulated controller
2 American Institute of Aeronautics and Astronautics
I/O Interfaces I/O interfaces are configured through custom blocks, supplied with RT-Lab. The user merely needs to add the blocks to the graphic model and connect the inputs and outputs to these blocks. The automatic code generator will map the model’s data flow onto the physical I/O cards.
Usually, simulation and control begins with models of hardware because the actual hardware is not yet available. Later, when it is available, the models of that hardware can be removed and the actual hardware connected. Thus, the hardware can be tested on-line as early as possible in the design process so that the real-time performance can be evaluated. Distributed simulation Sufficiently complex system dynamics, i.e. megasimulation, and/or sufficiently small step size will ultimately overwhelm even the fastest processor. Millisecond or faster cycles are not rare! The solution is parallel processing. RT-Lab implements parallel simulation in the form of a PC cluster, as shown in Figure 5.
simulated trajectory
virtual aircraft
controller
distributed simulator
input signals from sensors
node 1
control signals to actuators
node 2
input channels on communication card
output channels on I/O card
node n
Figure 5. Distributed HIL simulation Figure 6. Graphic tools for defining a simulation.
In keeping with the guiding principles, HIL is accomplished using commercial components. These are in fact simply PCs equipped with I/O boards.
Automatic Code Generation The high cost of software development and the potential for human error during coding are strong arguments for automatic code generation. RT-Lab code “understands” distributed, real-time simulation, and works with the standard Matlab or MATRIXX code generators RTW and AutoCode to enable parallel, real-time execution and physical I/O interface. RT-Lab automatically takes care of setting up all the communication, including realtime data exchanges between processors and interfacing with the real-time operating system, QNX™ 3 or RT-Linux4. This leaves the engineer free to work on the already difficult task of creating a proper mathematical model.
From simulation to rapid prototype Once the simulation has been completed, the same automatically generated code that was used for the HIL simulation may be deployed in real-life as the actual control system, simply by downloading into the flash memory of an embedded controller. This provides the ultimate in rapid prototyping, from simulation to functioning controller with a few mouse clicks.
Tools for Simulation Control The software described includes tools for simulation and control, automatic code generation, downloading and execution monitoring. The tools developed by Opal-RT are part of a package called RT-Lab. RT-Lab uses either of the popular Matlab/Simulink or MATRIXX/Systembuild tools packages as a front-end for model definition and viewing. The defined models become the source from which code can be automatically generated, manipulated and downloaded onto target processors for real-time or distributed simulation.
Computing Clusters RT-Lab supports distributed computing. Starting with Simulink or Systembuild models, complex simulations can be subdivided for execution on separate processors. Simply subdivide the models and add the inter-processor connection blocks supplied with RT-Lab.
™ 4
3 Trademark of QSSL RT-Linux support presently under development.
3 American Institute of Aeronautics and Astronautics
Opal-RT’s real-time simulation executes on a distributed system of standard PCs. No custom DSP, or proprietary boards are utilized. Typically, there is no performance penalty with this approach but there is a noticeable decrease in price [5]. Inter-PC communication options range from lowcost UDP/IP on Ethernet at 100Mb/s, through FireWire at 400Mb/s to high-performance cLAN™ 5 at 1.2GB/s. For ultra-high-performance, RT-Lab can make use of symmetric multiprocessing computers where the processors talk to each other directly through shared memory. Cluster size can be scaled to meet increased performance needs. Simply add processors and redistribute the simulation model at the graphic level.
Flight Simulator Application To demonstrate how RT-Lab can be used, a simplified flight simulator example is presented. Mathematical Model The starting point for any simulation is a mathematical model of the components that are to be simulated. With RT-Lab, the main concern of the engineer has moved from code writing to the creation of good real-time models. Figure 7 shows a sample MATRIXX model of an aircraft. The model is composed of 7 main blocks running at 10 ms: controls, aerodynamics, equations of motion, initialization, display, atmospheric data and engine. The presence of these discrete blocks suggests immediately that the model is amenable to decomposition for parallel execution on up to six nodes of a PC cluster (the init block is only executed once so it doesn’t need to be parallelized).
Execution RT-Lab supports two execution modes: as-fast-aspossible (which means what it says) and hard real time (which means the execution time of each simulation step is finitely bounded). Within both of these methods RT-Lab provides the interface to start, stop and pause the execution, as well as stepping through the execution. With large step sizes, say 100 milliseconds or greater, soft real-time can be achieved using a standard operating system such as Windows NT™ or Linux. With fine-grained step sizes, hard realtime execution requires the use of a real-time operating system such as QNX or RT-Linux. It is not unusual for HIL simulations to require submillisecond hard real-time step sizes, and RT-Lab as been used in hard real-time applications with step sizes as small as 22 microseconds6.
Decomposition and Parallelization The model is decomposed for use in RT-Lab from within the SystemBuild window. Decomposition is based on some simple rules: • all display elements are placed into a console block (SC_), • high-rate elements are placed in the master block (SM_), • all other elements are distributed into the slave blocks (SS_) as desired restricted only by the number of processors. Since we had one host-processor and a two targetprocessor cluster in our lab, we chose to decompose
Gear Forces 15 Y = 0
2 U
1 2 3 1 2 3 4
Controls
1
Aerodynamics 12
Equations of Motion 25
14 14
ALGEB
6
3
6 16
EXPR
6~19
12
5
7
5
0.01
39
1~30
0.01
0.01
2
18
INIT
Display 36
11
Atmospheric Data 34
34
2~23
16
9 0.01
1
19 3,5
1 0.01
6
16 4
0.01
Engine 21
34
4
™
6 6
17
6
25:27
3
Figure 7. Flight Simulation Model.
5 Trademark of Giganet Inc. Benchmark 22µs on a 350 MHz Pentium-based PC
4 American Institute of Aeronautics and Astronautics
5
9
4 5 0.01
20
the flight simulator application into 3 blocks (console, master and slave), as shown in Figure 8. The console block, SC_Animation, contains the display block as well as an OpComm block needed for inter-processor communication. The master block, SM_Controller, contains the initialization and controller blocks as well as the OpComm block; the 5 ms simulation rate has been arbitrarily selected to make this the high rate block. The remainder of the model has been placed in the slave block, SS_AC. This block contains two OpComms because it communicates over two different media. High-speed Firewire is used to share data with the master block at the 5 ms simulation rate, while standard Ethernet is used to update the display at a 10 ms rate (faster Ethernet rates are possible if desired).
Matlab or MATRIXX and uses RT-Lab to deposit the new model into the simulator. SC_Animation
SM_Gear 13
5
6
21
16 7 8 9 10 11 12 13 14 3 4 5 15
0.01
0.001
15
SS_Controls
SS_AC 23
1 2 3 4 5 6 7 8 9 10 11 12 13 14
22
1
2
15
1 4 5 3 0.005
SC_Animation
2 3 4 5 6
SM_Controls 13
23
Figure 9. Distributed Model with Landing Gear.
1
2
Visuals and other interfaces RT-Lab provides for distributed real-time dynamic simulation, and its open architecture allows thirdparty COTS geometry engines and visual rendering to be connected for out-the-window visuals. Because of the complexity of geometry, rendering and visual effects calculations, it is good practice to dedicate a separate PC for this purpose. Graphics cards developed for the video game market provide a low-cost, high-performance solution for visuals. Display can be through window-mounted monitors or commercial projectors. Virtual avionic instrument displays can also be provided using COTS solutions such as Labview™ 7, ALTIA™ 8or VAPS™ 9. Touch screen hardware can be installed over virtual switches to mimic toggles, flap levers or gear levers. This provides an inexpensive alternative to expensive hardware and gives the flexibility to easily reconfigure the simulator. Display computers linked with RT-Lab via TCP/IP on Ethernet give typical refresh rates greater than 30Hz. Figure 10 below shows the complete configuration including host processor, real-time PC simulation cluster, out-the-window and virtual-instrument graphics and HIL.
15
5
6
0.01
3 0.005 0.01
SS_AC 22
0.01
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 4 5
Figure 8. Distributed Flight Simulation Model.
Once the user has sub-divided the model into blocks, RT-Lab asks which blocks should be downloaded onto which target nodes in the PC cluster. RT-Lab then •generates code, •distributes to the various targets, •provides an interface for model execution and parameter manipulation. The result is an aircraft simulation running in parallel and in real-time. Reconfiguration In the course of the simulation study, the design engineers decided that they wanted to add a landing gear block to the model. In figure 9, the gear model is placed in its own block SM_Gear. This new block can share a processor or it can be mapped to its own dedicated processor. The choice depends on the size of the computations and the on real-time performance requirements. It is interesting to note that if external interfaces are fairly generic and one has models for different aircraft, then the simulator described can be used as a re-configurable simulator. To switch between aircraft, the user simply loads a different model into
Hardware-in-the-loop Interface to the real world via analog and digital input/output cards is a prerequisite for HIL simulation. RT-Lab’s open architecture allows a wide choice of PC-based cards to be used to talk to various COTS hardware such as pilot inceptors. Inceptor force feedback can then be controlled by the simulation. Programmable force feedback is ™ ™ ™
7
Trademark of National Instruments Trademark of Altia Inc. 9 Trademark of Virtual Prototypes 8
5 American Institute of Aeronautics and Astronautics
recommended for re-configurable applications or man-machine interface studies.
defined models become the source from which code can be automatically generated (using RTW or AutoCode), manipulated and downloaded onto target processors for real-time or distributed simulation. Once the simulation has been completed, the same automatically generated code that was used for the HIL simulation can be deployed in real-life as the actual control system, simply by downloading into the flash memory of an embedded controller. This provides the ultimate in rapid prototyping, from simulation to functioning controller with a few mouse clicks. Use of RT-Lab has been demonstrated for an aircraft simulator, starting with a MATRIXX model of an aircraft. With RT-Lab, it is also possible to have innovative solutions to new demanding problems. The system aims to maximize the speed at which new designs can be tested and implemented, while minimizing the effort of the engineers. From imagination to real-time, without ever having to write code.
real-time PC cluster (QNX) NT RT-Lab host
TCP/IP Ethernet NT
master block
QNX
controls block
QNX
gear block
QNX
FireWire or cLAN
Virtual-instruments graphics inceptor control
QNX
NT Out-the-window graphics projector
inceptors (HIL)
Figure 10. Simulator Configuration
RT-Lab can also communicate directly with other systems using standard protocols such as those standardized by ARINC. Complex configurations can be implemented. Flyby-wire flight control system can first be modeled on a simulation cluster. As components arrive they replace simulated components on a block-by-block basis.
References [1] L. Pollini and M. Innocenti, A Synthetic Environment for Dynamic Systems Control and Distributed Simulation, IEEE Control Systems Magazine, April2000, pp. 49-61. [2] J.C.Stasko, R.L Crandell, M.T. Dunn, N. Sureshbabu and W.Weber, The Versatile Hardware-in-the-loop Laboratory: Beyond the Ad-hoc Fixture, Proceedings 1998 American Control Conference, Philadelpia, PA, June 1998
Performance Enhancements Cost need not be a restriction on simulation complexity. With the constant increase in performance of PC’s and their lower price it is easy to justify continuous upgrades to ones computing cluster. It is also possible to have innovative solutions to new demanding problems. For example precise motor controls require very high frequency controllers. Magnetic flux calculations must be computed at step sizes < 50µs. Even more stringent, hard real-time I/O is often required for this class of problem. Opal-RT has proposed a solution using the latest in PC technology, multiprocessor PC’s. With a single board tandem PC sharing on board memory, one CPU can be used for extremely fast computations while the other does slower tasks like I/O and communication with other PC’s on the cluster.
[3] C.A.Rabbath, M.Abdoune, J.Belanger, Effective Real-Time Simulations of Event-Based Systems, Opal-RT Technologies Inc. White Paper, 1999. [4]
J.Ozard and H.Désira, Simulink Model Implementation on Multi-Processors under Windows-NT, Defense Research Establishment White Paper, 1999.
[5] C.A.Rabbath, Real-Time Simulation: Pentium vs. DSP, Opal-RT Technologies Inc. White Paper, 1999. About the Authors
Conclusions
Marco Papini. M.A.Sc., Senior Engineer, Aerospace Modeling and Simulation Services, Opal-RT. He has more than 15 years of experience in aeronautics including control law design, fly-bywire system simulation, aircraft flight simulation, flight mechanics and aerodynamics.
RT-Lab implements parallel simulation in the form of a PC cluster. The package includes tools for simulation and control, automatic code generation, downloading and execution monitoring. In keeping with the guiding principles, HIL is accomplished using commercial PC-based components. RT-Lab uses either of the popular Matlab/Simulink or MATRIXX/SystemBuild tools packages as a front-end for model definition and viewing. The
Paul Baracos. Ph.D., P.Eng., has been involved in simulation and control systems since 1976. He has degrees in Physics, Systems Design and Mechanical Engineering and is a member of the Manitoba Association of Professional Engineers. He joined Opal-RT as Director of Product in 2000. In his spare time he serves as chairman of the ISA Automation and Control Systems Committee.
6 American Institute of Aeronautics and Astronautics
