COOLFluiD: an open computational platform for multi-physics simulation and research A. Lani,∗ N. Villedieu,† K. Bensassi‡ L. Kapa,§ M. Vymazal, ¶ M. S. Yalimk and M. Panesi∗∗ The COOLFluiD (Computational Object Oriented Libraries for Fluid Dynamics) project, officially started in 2002, has lead to the creation of a world-class, extremely modular collaborative platform for high-performance scientific computing and multi-physics modeling. Target applications include Space Weather modeling, aeroacoustics, turbulence, aerothermodynamics for aerospace problems ranging from incompressible regime to hypersonics, steady and unsteady flows, etc. COOLFluiD features a unique collection of parallel numerical solvers for unstructured meshes, where each physical model (set of PDE’s, variable transformation, aerothermochemical properties, etc.) or numerical algorithm (space discretization, time discretization, coupling strategy, linear system solver interface, etc.) is enclosed in a separate module which is loaded dynamically at run-time. The paper presents a concise description about the main design strengths of the platform, an overview of current capabilities (in terms of numerical methods and physical models) and a gallery of the most challenging applications tackled so far. Particular attention is devoted to the computational modeling of aerothermodynamics and plasma flows, including atmospheric re-entry flows and numerical simulation of experiments in high-enthalpy facilities.
Nomenclature Ev Ep B0 B1
= = = =
electric field induced by the coils electric field produced by the currents in the plasma planetary dipole magnetic field variable magnetic field
I.
U ρ Φ Vref
= = = =
conservative variables density potential field reference velocity
Introduction
In order to tackle increasingly complex multi-disciplinary scientific and engineering problems, numerical tools have been rapidly evolving from single-application codes towards multi-physics and multi-domain frameworks, where different models can be coupled together to reproduce actual physical phenomena and experimental tests with higher fidelity. Some of the most popular platforms are reviewed hereafter. Overture is a set of object-oriented tools for solving CFD and combustion problems (mostly at low speed) with Finite Difference (FD) and Finite Volume (FV) methods on composite overlapping curvilinear grids with Adaptive ∗ Sen. Research Eng., Department of Aeronautics and Aerospace, Waterloosesteenweg 72, 1640, Sint Genesius Rode, Belgium;
[email protected] † Sen. Research Eng., Department of Aeronautics and Aerospace, Waterloosesteenweg 72, 1640, Sint Genesius Rode, Belgium;
[email protected] ‡ PhD Candidate, Department of Aeronautics and Aerospace, Waterloosesteenweg 72, 1640, Sint Genesius Rode, Belgium;
[email protected] § Sen. Research Eng., Department of Aeronautics and Aerospace, Waterloosesteenweg 72, 1640, Sint Genesius Rode, Belgium;
[email protected] ¶ PhD Candidate, Department of Aeronautics and Aerospace, Waterloosesteenweg 72, 1640, Sint Genesius Rode, Belgium;
[email protected] k Postdoc, Centre for mathematical Plasma-Astrophysics, KU Leuven, Celestijnenlaan 200b, 3001 Heverlee, Belgium;
[email protected]. ∗∗ Assistant Professor, University of Illinois at Urbana-Champaign, 306 Talbot Lab, 104 S. Wright St. Urbana, IL 61801;
[email protected]
1 of 25 American Institute of Aeronautics and Astronautics
Mesh Refinement (AMR) and moving components. LibMesh and Deal.II provide frameworks for solving compressible flows, multi-phase, shallow water, multi-scale material problems using arbitrary unstructured discretizations (FEM, FV, DG) with AMR techniques (hp-refinement). OpenFOAM is arguably the most successful open source CFD toolbox to date, counting on a large user-base across most areas of engineering and science, solving anything from complex flows with chemical reactions, turbulence and heat transfer, to solid dynamics and electro-magnetics. Because of its segregated iterative approach, each PDE is treated as a decoupled scalar equation, which unfortunately makes it only suitable for incompressible and low speed flows. OpenFOAM solvers are mostly based on standard FV schemes. The COOLFluiD (Computational Object Oriented Libraries for Fluid Dynamics) project,1–4 officially started in 2002, lead to the creation of an extremely modular collaborative platform for scientific computing and multi-physics modeling. The framework is written in C++, using advanced object-oriented techniques,5, 6 including design patterns, expression templates,7 combination of static and dynamic polymorphism. Parallelization is achieved through Message Passing Interface (MPI) and CUDA is used for computing on General Purpose Graphical Processing Units (GPGPU). So far, target applications have included aerothermodynamics (ATD) for aerospace problems, Space Weather modeling, electro-chemistry, aeroacoustics, turbulence, conjugate heat transfer, aeroelasticity, etc. COOLFluiD features an unrivalled collection of numerical solvers for unstructured meshes where each physical model or numerical algorithm is enclosed in a separate plugin library which can be loaded dynamically at run-time. The kernel defines the parallel data structure, abstract interfaces for models and algorithms (to be implemented by the plugins), key functionalities such as simulation control and parallel I/O. This situation is schematically depicted in Fig. 1, showing, as a concrete example, the simplified structure of the COOLFluiD aerothermodynamics (ATD) solver. Both a Finite Volume Method (FVM) and Residual Distribution Schemes (RDS) are available for ATD applications, both sharing the same models (e.g., thermo-chemical nonequilibrium equations, the interface to the MUTATION library providing aerothermochemical properties). A radiation module and a flow-radiation coupling infrastructure have also been recently developed within the ESA AMOD project, as discussed in Sec.V.
Figure 1. Schematic view of the COOLFluiD ATD solver.
The goal of this paper is to present a concise overview about some of the main contributions and results accomplished using COOLFluiD during ten years of multi-disciplinary and collaborative research. This will also offer the occasion to discuss in more details some of the main capabilities of the platform, including a wide range of applications for which a variety of physical models and numerical methods have been developed.
II.
COOLFluiD design
COOLFluiD is a component-based platform featuring a collection of dynamically linked libraries and application codes, organized in a multiple layer structure. Two kind of libraries are identifiable: kernel libraries and modules (plug-ins). The former supply basic functionalities, data structure and abstract interfaces, without actually implementing any model or algorithm, while the latter add effective simulation capabilities. The whole framework was developed according to a plug-in policy, based on the self-registration1, 2 and self-configuration techniques,3 which allow new components or even third party extensions to be integrated 2 of 25 American Institute of Aeronautics and Astronautics
at run-time, while only the API is exposed to the developers. From top to bottom of the COOLFluiD architecture, one finds, progressively, more functional layers: the kernel defines the interfaces, the modules implement them and the application codes can select and load on demand the libraries needed for an actual simulation. According to a simplified conceptual view, we can consider numerical solvers for PDEs to be made by three main building blocks: • mesh data representing a discretized view of the computational domain; • a physical model describing the phenomena under investigation; • a set of interacting numerical algorithms to preprocess, solve and postprocess the given problem. A general-purpose environment for solving PDEs should therefore provide generic support for tackling each of those three basic components. By treating them independently, as much as possible, it is possible to minimize tight coupling, to maximize the reusability of the individual components and, ultimately, to ease life for future developers. This is schematically depicted in Fig. 2 for COOLFluiD.
COOLFluiD
Physics
Numerics
Mesh Data
Input Mesh
CFD Simulation
Flowfield
Figure 2. COOLFluiD: from user-defined inputs to engineering analysis.
The following subsections will provide an overview of the actual object-oriented design solutions which are used in COOLFluiD in order to handle mesh data, physics and numerics. In all diagrams to be shown, graphic conventions from the Object Modeling Technique (OMT) are adopted. In particular, empty and filled arrowheads indicate inheritance and composition (acquaintance or aggregation) respectively, dashed lines stands for creation. A.
Mesh Data
Data creation, access, destruction needed by a numerical simulation is handled by MeshData, a facade object which encapsulates two kind of underlying structures: topological region sets and a data storage. 1.
Topological Region Sets
The computational domain can be considered as composed by topologically different regions which correspond to the TopologicalRegionSet (TRS) concept in COOLFluiD. Each TRS consists of a list of TopologicalRegions (TR), i.e. subsets of the domain ideally linked with a particular CAD representation. Each set of boundary surfaces on which a certain boundary condition has to be enforced can be interpreted as a different TRS, while the whole boundary can be broken into smaller patches, TR, according, for instance, to the surface definition in the CAD model. If we consider, as in the example in Fig. 3 (a), the
3 of 25 American Institute of Aeronautics and Astronautics
GEOMETRIC
Mesh Data
GEOMETRIC
Mesh Data
ENTITY BUILDER
TR
TRS
NODE
Boundary TRSs
CELL
SHAPE FUNCTION
FACE
SHAPE FUNCTION
GEOMETRIC ENTITY
TRS
ENTITY BUILDER
TR
STATE
CELL
SHAPE FUNCTION
FACE
SHAPE FUNCTION
GEOMETRIC ENTITY
NODE
STATE
Inner TRS
(a) Topological Region Sets
(b) Geometric Entities
Figure 3. Mesh data overview with focus on Topological Region Sets and Geometric Entities.
computational domain surrounding a spacecraft, FarField, Body, Flaps, Outlet can all be considered as different TRS’s. One could alternatively choose to pack Body and Flaps together to form a unique continuous TRS (e.g. Wall) consisting of two TR’s. Because of consistency and conveniency, not only the boundaries but also the interior of the domain is assigned to one or more TRS’s (e.g. Inner), according to the needs of the active numerical methods. The latter are typically applied on geometric entities (e.g. cells, faces), which represent the basic entities in the topological dicretization of the domain. A TRS is a proxy object that provides transparent access to the local and global connectivity for the aggregated geometric entities. In order to minimize the memory requirements, all those data are stored sequentially in unidimensional arrays (but logically bidimensional) which can be addressed by the client code exclusively through the TRS or TR interfaces. Geometric Entities. A GeometricEntity (GE) is a locally/globally indexed object defining an algorithmdependent agglomeration of degrees of freedom, both in geometric (Nodes or coordinates) and solution space (States or solution vectors), and by ShapeFunctions for the solution and geometric interpolation. Both Nodes and States are provided with global (inter-process) and local (intra-process) IDs, which are used to build connectivity information. As a result, each TRS (or TR) can be logically viewed as set (or subset) of GE in which each portion of the domain can be decomposed for computational purposes. GE’s are relatively lightweight objects which are constructed on-the-fly during the simulation from a small preallocated memory pool (depending on the local stencil requirements) by a GeometricEntityBuilder, as shown in Fig. 3 (b). Each spatial discretization can have more than one GE builder, depending on the algorithmic needs (e.g. for faces, for cells, possibly with different stencil data). 2.
Data Storage
Numerical algorithms typically make use of arrays of data deriving from the domain discretization, such as coordinates, state variables, normals, etc. As a result, genericity for allowed data types must be addressed. Moreover, in a scalable and safe design of the data handling, framework components should be able to share data without violating encapsulation. In a HPC environment, developers of new modules should be allowed to elect newly defined data types to be stored and handled efficiently in parallel communication. Ideally, local and distributed data should be treated uniformly from the developer’s point of view, and support for different communication strategies (e.g. message passing and shared memory) should be offered. The DataStorage object tackles all this complexity by behaving as a centralized archive storing key-value pairs, where numerical components can register and share both serial or distributed arrays of generic data types (values) by name (key). This situation is depicted in Fig. 4.
4 of 25 American Institute of Aeronautics and Astronautics
GEOMETRIC ENTITY BUILDER
Mesh Data
TR
TRS
CELL
SHAPE FUNCTION
FACE
SHAPE FUNCTION
GEOMETRIC ENTITY
DATA STORAGE NODE "nodes"
NODE
"states"
STATE
"normals"
NORMAL
...
...
STATE
Figure 4. Mesh data overview with focus on Data Storage.
B.
Physical models
In COOLFluiD, the implementation of all physical models, from the most basic to the most complex, is standardized and follows a predefined structure, namely the Perspective pattern. 1.
Perspective pattern
The interface for a generic physical model, which can, in principle, be hard to define, is broken into a granularity of independent Perspective objects,1, 5 each one offering a partial view of the same physics. To make an example, convective, diffusive, source terms can all be Perspectives with a certain abstract interface. As an example, Fig. 5 shows the diagram of a Perspective pattern applied to a generic advection-diffusion reaction model. The base PhysicalModel defines a very narrow abstract interface. ConcreteModel derives from PhysicalModel, implements the virtual methods of the parent class and defines another interface to which the concrete Perspective objects (and only those) are statically bound. This other interface offers data and functionalities which are typical of a certain physics, but invariant to all its possible Perspectives. PHYSICAL MODEL
Concrete Convective Term CONVECTION DIFFUSION REACTION
CONVECTIVE
DIFFUSIVE
REACTIVE
VARSET
VARSET
VARSET
Concrete Diffusive VarSet
Concrete Reaction VarSet
Concrete Convective VarSet
Concrete Diffusive Term Concrete Reaction Term
Concrete Physical Model
Figure 5. Perspective pattern applied to a advection-diffusion-reaction physical model.
The resulting pattern lets the numerical client code make use of the physical model through an abstract layer, dynamically enlargeable if required, given by a number of Perspective objects which make transparent 5 of 25 American Institute of Aeronautics and Astronautics
use of the ConcreteModel. The basic goal of this approach is that, as long as the abstract interfaces are untouched, one can virtually replace one physical model (or a subset of it) with another one and still have a working numerical solver. To this end, only suitable boundary or initial conditions will still typically need to be adapted or implemented on the side of the solver. Variable Sets. A variable set, VarSet, is a Perspective that decouples the usage of a physical model from the knowledge of the used physical variables. This, for instance, allows the client code to solve equations formulated in conservative variables while performing intermediate algorithmic steps (e.g. residual calculation, solution interpolation or limiting or update) in other more convenient variables (e.g. primitive, characteristic). In order to deal with a system of convection-diffusion-reaction equations, three variable sets classes can be defined: ConvectiveVarSet, DiffusiveVarSet, ReactiveVarSet. Each VarSet has a different interface. For instance, computation of convective fluxes, flux jacobians, eigensystem are provided by a ConvectiveVarSet, while calculation of diffusive fluxes and transport properties belong to a DiffusiveVarSet interface. Variable Transformers. Another useful Perspective is the VariableTransformer. This allows the client code to apply matrix similarity transformations with the form ∂U/∂V between variables or to compute one set of variables from another U(V) analytically (e.g. from primitive to conservative and viceversa). The combined use of VarSets and VariableTransformers gives the freedom to use different variables to update the solution, compute the residual, linearize jacobians, distribute residuals, without requiring any modification in the client numerical algorithms which are typically unaware of the actual physics. The support for variable manipulation and variable independent algorithms which is offered by COOLFluiD represents a key feature for the power of COOLFluiD in handling the most diverse and complex models. C.
Numerical algorithms
The solution of PDE’s requires the implementation of different numerical techniques that deal with time discretization, space discretization, linear system solving, mesh adaptation algorithms, error estimation, mesh generation, etc. In a typical object-oriented design, each one of these simulation steps is enclosed in a separate object (or polymorphic hierarchies of objects) and different patterns are applied to make them all stick together and interact. In COOLFluiD, any class of algorithms can be implemented by means of a uniform high-level structural solution, named Method-Command-Strategy (MCS) pattern. The latter has proven to ease the integration of new algorithms, but also the implementation of different versions or parts of the same algorithm, while looking for optimal solutions and/or tuning for run-time performance. 1.
Method Command Strategy pattern
The MCS pattern1, 8 provides a uniform way for implementing numerical algorithms and to customize them according to the specific needs. As sketched in Fig. 6, the pattern is structured as follows: • BaseMethod defines an abstract interface for a specific type of algorithm (e.g. LinearSystemSolver, ConvergenceMethod, SpaceMethod, ErrorEstimator, . . . ); • ConcreteMethod implements the virtual functions of the corresponding parent class by delegating tasks to ad-hoc Commands, see9 ;10 • Commands share ConcreteMethodData, i.e. a tuple providing access to polymorphic receivers, so-called Strategies;10 • Strategies cooperate with concrete Commands by taking care of some specific parts of the overall algorithm. Three levels of abstraction and flexibility are identifiable in this pattern: BaseMethod, Command and Strategies can all be overridden, allowing one to implement the same task, at the corresponding level, in different ways. This kind of behavioral modularity allows the developers to easily re-implement or tune components (Methods, Commands or Strategies), and gives them the freedom to move code from one layer to another, according to convenience, taste or profiling-driven indications. The fast-path code, critical for the overall performance,
6 of 25 American Institute of Aeronautics and Astronautics
BaseMethod
action1() action2()
ConcreteMethodData
getStrategyA() getStrategyB() getStrategyC()
ConcreteMethod
action1() action2()
Command
execute() Command
execute()
... Action1−>execute() ...
StrategyA
Concrete StrategyA
StrategyB
Concrete StrategyB
StrategyC
Concrete StrategyC
Action1
execute() Action2
execute()
Figure 6. Diagram of the Method-Command-Strategy pattern.
can be wrapped inside Commands or Strategies and it can be substituted with more efficient implementations without implying changes in the upper layer. In COOLFluiD, managing the collaboration between different numerical methods is eased by the fact that Commands and Strategies can create their own local or distributed data and share them with other Commands defined within other Methods, by making use of the DataStorage facility presented in section 2. Moreover, the Perspective pattern described in section B can be used in combination with MCS, at the Strategy level, to dynamically bind a numerical algorithm to the physical model. Interchangeability of Methods, Commands, Strategies can be facilitated and maximized by making them self-registering 1, 2 and self-configurable objects.3 A deeper analysis of the MCS pattern and exhaustive implementation details are provided in.3, 8 Among the possible BaseMethods that can be identified in a multi-physics simulation, and among those we have actually implemented in COOLFluiD, we consider now a few examples in order to show the concrete applicability of MCS. Example of SpaceMethod: Finite Volume. As an illustrative example for a BaseMethod, we choose a SpaceMethod, taking care of the spatial discretization of a given set of PDE’s. Figure 7 shows a class diagram of a possible concrete space method, namely a FVM Method. For didactic purposes, the interface of the SpaceMethod has been simplified and reduced to two (pure) virtual functions, applyBC() and computeRHS(). The actual implementation of the latter is delegated by the corresponding member functions in FVM Method to two specific Commands, denominated WallBC and ComputeRHS. SpaceMethod
Physical Model
applyBC() computeRHS()
FVM_Method
applyBC() computeRHS()
Command
execute() Command
execute()
FVM_MethodData
getVarSet() getExtrapolator() getFluxSplitter() ... WallBC−>execute() ...
Concrete Physical Model
VarSet
Concrete VarSet
Extrapol
Concrete Extrapol
FluxSplit
Concrete FluxSplit
WallBC
execute() ComputeRHS
execute()
Figure 7. Diagram for a MCS pattern applied to a simplified FV space discretization.
7 of 25 American Institute of Aeronautics and Astronautics
Those Commands share access to a tuple class, FVM MethodData, which groups together polymorphic Strategy objects whose task consists in implementing part of the algorithms defined inside the Commands. In our oversimplified view, we consider three Strategies: (1) Extrapol that extrapolates the solution with the chosen degree of accuracy (first, second order), (2) FluxSplit which discretizes the convective fluxes of the given set of equations, (3) VarSet which provide a dynamic binding of the numerical algorithm to the physics. Within this dynamic design structure, each part of the FV method can be customized at will and different extrapolation algorithms, schemes, BCs, physical models, CPU- or GPU-based algorithms, etc. can be provided and automatically plugged-in. Other space discretizations such as FEM, Discontinuous Galerkin or Residual Distribution have been implemented in a similar way as subclasses of SpaceMethod. Example of Collaboration: ConvergenceMethod and LinearSystemSolver. We consider now an example of interaction between two different numerical methods. Figure 8 shows the collaboration between two methods: ConvergenceMethod, responsible of the iterative procedure, and LinearSystemSolver (LSS). In this case, BackwardEuler, an implicit convergence method, delegates polymorphically the solution of the resulting linear system to PetscLSS, which interfaces the PETSc library.11 Convergence Method
Backward Euler
LinearSystem Solver
Backward Euler Data
Petsc LSS Data
PetscLSS
Command
Setup
Setup
Command
Command
Unsetup
Unsetup
Command
Command
Update Solution
Solve System
Command
MeshData
Figure 8. MCS pattern applied to two interacting modules: a Backward Euler convergence method and a Petsc linear system solver.
ConvergenceMethod declares three virtual functions, whose implementation is delegated in BackwardEuler to corresponding Commands, namely Setup, Unsetup and UpdateSolution, all sharing the data defined in BackwardEulerData. LinearSystemSolver also declares three (pure) virtual functions, and PetscLSS, the derived class, assigns their implementation to Setup, Unsetup and SolveSystem respectively, all having access the some common data provided by PetscLSSData, such as, for example, references to the (parallel) Petsc matrix and the (parallel) Petsc vectors involved in the solution of the linear system. In PetscLSS only the PETSc Commands can make direct use of PETSc11 objects like matrices, vectors or KSP (Krylov SPace solver), which are aggregated by PetscLSSData. The knowledge of a specific linear system solver, PetscLSS in this case, is not needed anywhere in the numerical modules, as long as the corresponding abstract interfaces are properly defined. As represented in Fig. 8, all the involved Commands have acquaintance of MeshData, which provides access to DataStorage, and can therefore access data arrays by name and type, as explained previously. While each ConcreteMethodData (e.g. PetscLSSData, BackwardEulerData) allows for intra-Method sharing of ConcreteMethod-dependent data among Commands, MeshData is the vehicle for inter-Method data exchange. D.
Parallelization
COOLFluiD features parallel I/O capabilites both for reading and for writing files in its own native format (CFmesh). Mesh decomposition is performed by a cell connectivity-based algorithm from Parmetis12 during the mesh reading phase.
8 of 25 American Institute of Aeronautics and Astronautics
1.
Overlap region
Once all mesh elements have been distributed to the corresponding target processor, an overlap region is built: it consists of a set of elements to be shared among different partitions, in which the nodes and states are not all updated by the same processor. A typical situation is depicted in Fig. 9, where, after a cell-wise partitioning, a 1-layer overlap region has been built: it includes all the vertex neighbors of the local cells attached to the partition boundary in each process Overlap Region
Partition B Partition A
owned element
partition interface
overlap element
data synchronization
owned node
ghost node
sending node
receiving node
Figure 9. Example of 1-layer overlap region and data synchronization. Unknowns (states) are assumed to be placed at the nodes.
The parallel algorithm allows for constructing an arbitrary number of overlap layers, as required by the actual space discretization method. For example, algorithms with compact stencils (e.g. FEM, RDS) require one layer of overlap, a cell-centered finite volume method requires two layers of overlap if a least square reconstruction13 is applied, but can require three layers with a MUSCL extrapolation14 for structured grids. Figures 10(a) and 10(b) show the partitioned surface of a 3D unstructured mesh for the ESA EXPERT vehicle geometry when one or two layers of overlap are selected.
(a) Example of 1-layer overlapamong three partitions
(b) Example of 2-layer overlap among three partitions
Figure 10. Partitioned surface mesh for the EXPERT vehicle with 1-layer overlap.
9 of 25 American Institute of Aeronautics and Astronautics
2.
CPU/GPU hybrid computing
COOLFluiD data structure has been recently upgraded to perform calculations on latest Graphics Processing Units (GPU) architectures, by integrating some CUDA bindings into the C++/MPI kernel code. In particular, the Finite Volume core algorithm has been re-written to perform the full residual assembly (both for explicit and implicit time-stepping cases) on GPUs and tested on some MHD applications. Thanks to the extremely modular software architecture, the porting to GPU only affected very limited portions of the code. In particular, two new numerical Commands (see section 1) for the assembly of all fluxes and flux jacobian matrix entries have been implemented. Boundary conditions and time marching (including solution update) are kept outside the GPU. Data transfers between CPU and GPU memory has been minimized by using allocation of critical data on pinned memory. The porting to GPU has lead to an encouraging 30X (60X in single-precision) speedup for explicit 3D calculations and to a 1.2X to 5X speedup for implicit cases (depending on the chosen PETSc preconditioner and on the number of Krylov steps per iterations). Future efforts will focus on optimizing the hybrid MPI/CUDA parallelization strategy which is for the moment rather basic: a 1-to-1 correspondence between CPU cores and GPU devices has been assumed so far. E.
Summary of COOLFluiD capabilities
COOLFluiD includes a collection of multidimensional (1D/2D/3D) parallel solvers for unstructured meshes, featuring: • multiple space discretizations: cell-centered FV,2 RDS,2, 3, 15, 16 Spectral FV (SFV) and FD (SFD),17, 18 DG, FEM;3, 4, 19 • multiple time steppers: explicit (n-th order Runge Kutta) and implicit (3 point-backward, CrankNicholson, time-limited) schemes; • multiple interfaces to linear system solver packages: PETSc, Trilinos, Pardiso, etc.; • compressible and incompressible flows; • RANS turbulence models (k − ω, BSL, SST, Spalart-Almaras), LES and DNS; • multiple ATD models:2 thermo-chemical equilibrium and nonequilibrium, multi-temperature and Collisional Radiative (CR) models,20 Inductively Coupled Plasma models;21 • Magneto Hydro Dynamics (MHD) models for Space Weather applications;22 • particle tracking, Monte Carlo algorithm for radiation transport, flow-radiation coupling;23 • coupling algorithms for multi-physics and multi-domain simulations;4 • Arbitrary Lagrangian Eulerian (ALE) schemes and robust mesh deformation algorithms; • Hybrid MPI / CUDA parallelization for heterogeneous HPC computing.
III.
Atmospheric re-entry of space vehicles
Several state-of-the-art and more advanced models have been integrated in COOLFluiD for simulating hypersonic re-entry flows: standard multi-temperature models, Local Thermodynamic Equilibrium (LTE) with fixed and variable elemental fractions,2, 24, 25 full and reduced electronic-specific Collisional Radiative (CR) models.20, 26, 27 In particular, an extensive validation of multi-temperature models (e.g., 11-species Park’s model for air) is presented in2 for the Stardust Sample Return Capsule case. COOLFluiD has provided a major contribution to the aerodynamic database for the ESA BLAST concept mission. In this case, 18 3D configurations were simulated, ranging from M∞ = 1.3 to M∞ = 15, with angles of attack α varying from 0o to 40o . Different models were used: non-reactive perfect gas up to M∞ = 8 to thermochemical nonequilibrium models (5-species two-temperature model for air with Park’s rates) with radiative equilibrium at the wall for the case at the highest Mach.
10 of 25 American Institute of Aeronautics and Astronautics
(a) Computational mesh for the (b) Roto-translational temperature (c) Vibrational temperature field ESA BLAST capsule (4,363,072 field around ESA BLAST capsule at around ESA BLAST capsule at hexahedral cells). M∞ = 15 and α = 25o . M∞ = 15 and α = 25o .
The 3D computational mesh used for this analysis is shown in Fig. 11(a) and featured 4,363,072 hexahedral cells. Figures 11(b) and 11(c) show the flowfield in terms of roto-translational and vibrational temperatures respectively in the case at M∞ = 15 with α = 25o . It is particularly interesting to notice the strong effect of thermal nonequilibrium in the expanding flow region (e.g., passed the aft), where the Tv appears to be frozen at a much higher value than the roto-translational temperature. Small differences between the temperature appear, instead, near the blunted nose of the capsule, where the flow is almost in thermal equilibrium. The results of a more challenging 3D simulation of the ESA EXPERT vehicle at M∞ = 14 is shown in Figures 11(d) and 11(e). The computational mesh was composed of 3,840,453 hexahedral cells. The same thermochemical nonequilibrium model as in the BLAST case was used also for this simulation, imposing a given temperature surface distribution coming from material analysis provided by Thales Alenia Space (as can be seen in the vibrational temperature contours on the vehicle surface in Fig.11(e)), isothermal condition on the metallic cone and super-catalycity on the ceramic nose. The numerical analysis of a FIRE II trajectory point (at t=1634 s) is shown in Fig. 11: pioneering results with a complete (116 equations, 1D) and with a reduced (18 equations, 2D axisymmetric) electronic-specific CR model are compared in terms of population versus energy levels,26 showing a promising good agreement in predicting the departure from Boltzmann’s distribution. This was the first application of CR models to a realistic 2D simulation.
(d) Computational mesh for the ESA EXPERT vehicle (3,840,453 hexahedral cells).
(e) Computed Mach and Tv field around ESA EXPERT vehicle at M∞ = 14 and α = 0o .
11 of 25 American Institute of Aeronautics and Astronautics
22
10
21
Population/degeneracy
10
20
ABBA: Full CR Reduced CR 2D
10
19
10
18
10
17
10
16
10
15
10
14
10
13
10 0
2
4
6
8
10
12
14
Energy Levels [eV] Figure 11. Degeneracy predicted by ABBA CR-116 (1D) and reduced CR-18 (2D) for FIRE II (t=1634 s).
IV.
Experiments in high-enthalpy facilities
COOLFluiD can simulate experiments in high-enthalpy facilities in regimes ranging from incompressible to hypersonic, as described hereafter for three specific cases. A.
Modeling of experiments in the VKI Plasmatron
An Inductively Coupled Plasma (ICP) model,21 coupling chemically reacting (LTE) Navier-Stokes2, 28, 29 with electromagnetic induction equations, has been developed to simulate experiments in the VKI Plasmatron wind-tunnel in argon, air and CO2 . The result of one of such simulations with air as gas, including plasma torch, test chamber and TPS sample, is presented in Fig. 12.
Figure 12. Plasmatron experiment versus simulation (total molar fraction of charged particles is shown.)
In ICP wind tunnels, the gas injected into the facility is heated by an external inductor, so that the flow reaches very high temperatures in the torch (∼ 10, 000K). The equation of the electric field induced by the
12 of 25 American Institute of Aeronautics and Astronautics
torch reads: 1 ∂ ∂2E + 2 ∂x r ∂r
∂E r ∂r
−
nX coils E − iωµ σE = −iωµ σI δ (r − ri ) 0 0 c r2 i=1
(1)
where the electric field E is a complex variable. If we consider that the total field is equal to the sum of the electric field induced by the coils Ev and the electric field produced by the currents running into the plasma Ep , namely E = Ev + Ep , Eq. (1) can be split into two different equations: nX coils Ev ∂Ev ∂ 2 Ev − = −iωµ I δ (r − ri ) r + 0 c ∂r ∂x2 r2 i=1 1 ∂ Ep ∂Ep ∂ 2 Ep − 2 = iωµ0 σ (Ev + Ep ) r + r ∂r ∂r ∂x2 r
1 ∂ r ∂r
(2)
(3)
Equation (2) can be solved analytically, whereas the solution of Eq. (3) is more difficult to be obtained, since an electromagnetic field model is needed to compute Ep . The Navier-Stokes equations and the electromagnetic field equations are coupled and solved together, at the same time step. The torch creates an electric field that interacts with the ionized plasma, affecting its movement by means of the Lorenz force. Besides, the electromagnetic field produces also changes in the flow, increasing its temperature because of the Joule effect. On the other hand, the temperature changes influence the electric conductivity of the air, meaning that, in Eq. (1), the induced electric field will also be affected by the temperature changes. To sum up, these two systems of equations are coupled in both directions: the solution of the electric field system affects the Navier-Stokes system and vice versa. A detailed explanation about the modeling and numerical discretization of the Lorentz force and the Joule effect can be found in.21 B.
Double cone experiments
The first Author has been involved in the NATO RTO Task Group 43 (topic no 2),30 focusing on a further assessment of CFD for the specific issue of shock interactions and control surfaces in nonequilibrium flows. A double cone with semi angles of 25o and 55o and a base diameter of 10.3 inches was selected as one of the testcases. Extensive experimental studies have been conducted for this configuration in the LENS I shock tunnel at Calspan University of Buffalo Research Center (CUBRC) and measured surface pressure and heat flux data were available. In particular, one of the simulations (M∞ = 11.5, 2-species nitrogen flow), whose nominal conditions for the selected high enthalpy (9.17 MJ/kg) are listed in Table 1, was performed with COOLFluiD by means of an innovative Contour Residual Distribution (CRD) vertex-centered numerical solver, which has been extended for tackling axisymmetric nonequilibrium flows,31–33 using a standard 2temperature model. ρ∞ [kg/m3 ] 0.001468
U∞ [m/s] 3849.3
T∞ [K] 268.7
v T∞ [K] 3160
yN 0
yN2 1
Twall [K] 294.7
Table 1. Free stream conditions for double cone (run 42).
Figure 13(a) shows the overall flowfield in terms of Mach number contours and isolines, while Fig. 13(b) compares our results with those from all other participants to the workgroup, demonstrating an excellent performance of the novel numerical method on a relatively coarser (up to 4X) unstructured grid made of 131,584 triangular cells. C.
Modeling of experiments in the VKI Longshot
The VKI Longshot free piston tunnel (schematically depicted in Fig. 13(c)) is a short duration facility that can be operated with either nitrogen or CO2 and it is designed to reach high Reynolds number hypersonic flows. It is essentially based on two tubes (driver and driven tubes) and a nozzle which discharges into a test chamber. When a valve opens, a piston is released in the driver tube and travels at high speed through the driven tube, ending up compressing the gas in the reservoir of the nozzle through which the gas expands rapidly and reaches the model in the test chamber at high Mach (∼ 14 with a contoured nozzle). The dynamics of the piston in the tube is quite complex and involves shock reflections in front of 13 of 25 American Institute of Aeronautics and Astronautics
(a) Mach number contours/isolines in Run 42 con- (b) Surface heat flux on double cone: COOLFluiD ATD ditions. (RDS) vs. experiments and FVM solvers.
(c) Schematics of the Longshot free piston tunnel.
(d) Roto-translational and vibrational temperature distributions inside the nozzle.
the piston, rarefaction waves at the rear and a sophisticated system of valves and diaphragms. Pioneering approaches to model this kind of facility with COOLFluiD in operational conditions have started, including the steady expansion in the nozzle,34 the unsteady simulation of the expansion in the nozzle plus the testing over a heat probe placed at the exit,35, 36 the unsteady simulation of the piston dynamics inside the driven tube by means of an innovative multi-domain Arbitrary Lagrangian Eulerian (ALE) approach described in.37 Figure 13(d) shows the temperature distributions along the nozzle with and without including thermochemical nonequilibrium effects (freezing of the vibrational temperature).34 Figures 13(e) to 13(g) present the comparison between computed and measured time-dependent stagnation point heat flux on a heat probe, resulting from three implicit time accurate simulations at three different Reynolds numbers in non-reactive and laminar perfect gas conditions. The simulations have only been performed for the effective duration of the test, i.e. the first 15 ms of the actual experiment, and appear to match the data pretty well.
14 of 25 American Institute of Aeronautics and Astronautics
(e) Computed vs. measured stagnation heat flux on the probe at high Reynolds number.
(f) Computed vs. measured stagnation heat flux on the probe at medium Reynolds number.
V.
(g) Computed vs. measured stagnation heat flux on the probe at low Reynolds number.
Radiation transfer
An innovative Monte Carlo algorithm for computing the radiative transport in loosely coupled flowradiation aerothermodynamics calculations has been developed in COOLFluiD.23 The method allows to solve the Radiative Transfer Equation (RTE) by making use of a (1) particle tracking algorithm which relies solely upon vectorial operations and properties, particularly suitable for 2D and 3D unstructured meshes with arbitrary element types; (2) a Radiative Energy Absorption Distribution (READ) procedure, which associates each geometrical entity (cell or face) to a set of other entities which individually absorb a fraction of its energy particles. 0.8
0.6
q / σT
4
Analytical solution Monte-Carlo method
0.4
0.2
0 (h) Infinite slab testcase: computed heat flux (300 photons/cell) vs. analytical solution.
0.2
0.4
rα
0.6
0.8
1
(i) Infinite cylinder testcase: computed heat flux (32 photons/cell) vs. analytical solution.
Once a statistical pattern is computed via the ray tracing, it can be reused during the simulation and the Monte Carlo method reduces to sending variable energy packets to the predetermined destination elements at each coupling iteration. The algorithm has been validated on simple cases for which an analytical solution is available, such as an infinite slab and a infinite 3D cylinder. Details for those testcases are provided in.23 Figures 13(h) and 13(i) show a very favorable comparison between the computed and the analytical solutions for both cases, using a relatively limited number of photons per cell. In the slab case, a 3D mesh with 100x1x100 cells and 300 photons per cell have been used, 16x64x16 cells and 32 photons have been used for the cylinder case. Validation of the flow-radiation coupling for the conditions corresponding to points in the re-entry trajectory (at speeds up to 11Km/s) of the FIREII flight experiment is ongoing.38
VI.
Aeroacoustics modeling
Numerical simulation of noise phenomena is a very challenging problem. Sound production and propagation is an unsteady process. Sound waves propagate coherently over long distances with very low attenuation
15 of 25 American Institute of Aeronautics and Astronautics
due to viscous effects. On the other hand, even a very loud flow radiates only a small fraction of its total energy as sound which means that the perturbations to the mean flow are very small. Therefore, the numerical resolution in order to capture the sound propagation has to be very high39 . Various methods have been applied successfully to propagate noise in homogeneous and inhomogeneous background flows40 , however the trade-off between accuracy and computational cost still leaves room for further investigations on more efficient and robust solvers. A possible approach to encounter these requirements is to treat separately noise production from noise propagation. COOLFluiD provides a module that deals with the discretization of Linearized Euler Equations (LEE) by multidimensional upwind RDS.15 LEE are applied to model the propagation of sound in the domain where no source of sound is present and where the analogy methods such as Ffowcs-Williams can not be used because of gradients in the mean flow. The equations derived by Bailly et al.41 are used for inhomogeneous mean flow, written in conservative form: 0
0 0 0 0 ∂ ∂ ∂ρ 0 + (ρ0 uj + ρ u0j ) + { (ρ uj + ρ0 uj )} = 0 ∂t ∂xj ∂xj 0
0 0 0 0 ∂ ∂u0i ∂ρ0 ui + (ρ0 ui u0j + p δij ) + (ρ0 uj + ρ u0j ) + ∂t ∂xj ∂xj 0
0
0 0 ∂ρ ui ∂ ∂u0i 0 0 0 0 { + (ρ ui u0j − ρ0 ui u0j ) + (ρ uj − ρ0 uj )} ∂t ∂xj ∂xj 0 0 ∂ 0 0 (ρui uj − ρui uj )} = {− ∂xj
(4)
0
0 0 0 ∂u0i 0 ∂p0 ∂ ∂p + (γp0 ui + p u0i ) + (γ − 1)(p − ui ) ∂t ∂xi ∂xi ∂xi 0
0
0
+{ui
0
0 0 ∂u ∂u ∂p 0 ∂p i − ui + γ(p − p0 i )} = 0 ∂xi ∂xi ∂xi ∂xi
The terms subscripted with 0 are time-averaged mean flow quantities that are known functions (available for example from a RANS calculation). The variables with prime are the fluctuating perturbations, i.e. the unknowns. The overbar denotes time averaging. Bailly et al.41 reported that growing instability waves can be excited by source terms in LEE through the mean shear. In order to prevent the development of linear instability waves these terms are handled separately from the fluctuating derivatives. The terms containing multiplication of fluctuations are considered of second order and are neglected (in {}). The requirements on the numerical schemes in case of discretization of sound propagation are rather restrictive. In order to propagate waves over long distances with negligible dissipation and without phase error, a consistent, stable, and convergent high-order scheme is not enough to guarantee a good quality for the numerical wave solution42 . Indeed, wave propagation involves the interplay between space and time. So, to be able to compute a wave solution accurately it is not sufficient to have a good approximation of the spatial derivatives alone or of the time derivatives alone. Both of them must be well approximated in a related way as dictated by the dispersion relation of the original partial differential equation, otherwise the wave modes may become numerically coupled. RDS share properties of both FEM and FV. In particular, the schemes used here are multidimensional upwind which make them very attractive because of their low cross-dissipation.43 To get the same accuracy and dispersion/dissipation relation in space and time a coupled space-time discretization method is used in COOLFluiD.44 An other issue in aeroacoustic simulations is imposing subsonic outlet boundary conditions for compressible flow simulations, since spurious reflections are occurring if there are unsteady disturbances leaving the domain. Most of the non-reflecting boundary conditions can be found in the literature performing very well if only acoustic perturbations are considered, but they are less effective if there are vortical structures passing through the boundaries. In the literature these spurious waves due to unsteady perturbation passing though the boundaries are called reflection. However, if a vortical structure is leaving the domain, the acoustic waves propagating back to the solution domain are rather created. In this sense, the original characteristic equations are modified such that the source terms responsible for these spurious waves could be identified and deactivated. In such a way, the vortical disturbances can pass the boundaries and therefore no more buffer zone is needed.45 To demonstrate the high accuracy of wave propagation and the previously described boundary condition, let’s consider an acoustic benchmark case designed to test the effectiveness of radiation boundary conditions, 16 of 25 American Institute of Aeronautics and Astronautics
inflow and outflow boundary conditions. The initial value problem46 consists of an acoustic pulse centered at the origin and a combined vortex/entropy pulse placed at (x, y) = (67.0, 0.0). The mean flow is uniform with Mach number Mx = 0.5, My = 0.0 and the time step is ∆t = 0.3 as a consequence of past-shield condition44 . The computational domain extends from −100 ≤ x, y ≤ 100 embedded in free space and the boundary is discretized by 201 × 201 nodes, and the interior domain with unstructured triangles. The perturbation velocities, density and pressure are normalized by c0 (ambient speed of sound), ρ0 (ambient density) and ρ0 c20 respectively. The two pulses meet each other just at the moment when they reach the right boundary. Applying the newly developed boundary condition the reflections are reduced to less than 1% all-over the domain (Figure 13).
(j) Hedstrom’s BC
(k) Proposed BC
Figure 13. Density contours of the solution of the benchmark problem at t=90. 10 iso-contours defined by a geometrical ratio of 10 from -0.001 to 0.001.
VII.
LES modeling
Compressible Large-Eddy Simulation (LES) was carried out to study the aerodynamics of simplified slat and wing configuration, with a particular emphasis on the flow field in the cove region. The upper trailing edge was kept blunt in order to resolve the vortex shedding towards the wing leading edge. The unsteady flow in the slat cove appears to be very complex. The main shear layer impinges to the slat pressure side. Some of these vortices travel through the gap while others stays inside the recirculation area. Analysis of the unsteady data revealed a periodic impingement of vortices to the wing leading edge. Due to the impingement the boundary layer turns to turbulent, but the transition point oscillates accordingly. The flow conditions of the simulation correspond to a Mach number M = 0.146 and Reynolds number Re = 1.02 · 106 . The angle of attack of the mean flow is 18◦ . The chord of the wing is 300mm, while the slat chord is 100mm. The geometry is a quasi-2D geometry, a 2D geometry is extruded in the spanwise direction. The trailing edge of the main element and the lower trailing edge of the slat were considered as sharp edge, while the finite thickness of the upper trailing edge of the slat was kept.In order to capture the dynamics of the incoming boundary layer a dense boundary fitted mesh was generated. The maximum y + is inferior to 3 in the whole domain. Depending on the streamwise location, the spanwise length scale is expected to vary significantly over the computational domain (i.e. the spanwise length scale is expected to be larger in the free shear flow than in the trailing edge regions). For accuracy reasons only FEM-based methods, such as the currently considered RDS, can use varying mesh resolution in the spanwise direction. A comprehensive mesh dependence study is currently not feasible for 3D time accurate simulations. A preliminary steady study was conducted over 3 different mesh resolutions in order to approximate the necessary grid resolution, however, in the current case the 2D flow topology is greatly different from the 3D one. So, finally, the recommendations and experiences described in the literature was used to construct the mesh.47, 48 Despite the low Mach number, the compressible LES RDS solver was chosen for performing this simulation 17 of 25 American Institute of Aeronautics and Astronautics
in order to be able to resolve the acoustic-flow coupling, which was thought to be present in the cove region. The Smagorinsky model was used to model the unresolved scales. Figure 14 represents the flow qualitatively with the Q iso-structures that are displayed in the cove region. The main recirculation bubble is bounded by the shear layer originated from the lower slat trailing edge. Some elongated structures can be seen downstream of the lower trailing edge, but rapidly the shear layer turns to a complex group of 3D structures embedded in each others.
Figure 14. Q iso-structures above and below the wing colored by vorticity magnitude.
The shear layer originated from the lower trailing edge impacts on the slat wall at the cove side. Some of the structures got trapped by the recirculation bubble and travel along the slat cove wall, while others are convected by the flow through the gap between the wing and the slat. In the upper part of the cove the flow cross section reduces so the mean velocity increases. Due to the velocity gradient in the gap between the main element and the slat upper trailing edge, the turbulent structures are elongated. The upper trailing edge of the slat has a finite thickness. This blunt trailing edge induces a vortex shedding. The shear layer seems to roll-up into discrete spanwise vortices almost immediately downstream of the upper trailing edge. The quasi-2D vortices soon turn to three-dimensional and streamwise vortex streaks appears. In Figure 15 it can be seen that the upper and lower boundary layer arriving to the slat upper trailing edge is very different from each other. At the cove side a very thin, turbulent boundary layer arrives to the upper trailing edge, while at the other side a thicker, laminar boundary layer is developed. The higher momentum of this turbulent boundary layer cause the shear layer turn away from the wing suction side. Examination of several images in time showed no significant flapping of this shear layer. The plot of the instantaneous Schlieren-like view in the mid-plane section shows a periodic bouncing of vertical package into the noise suction side of the main wing. The high energy of this package triggers the boundary layer to turn to turbulent. Indeed, the boundary layer around the wing leading edge seems to be laminar, up to the point where the shear layer originated from the upper trailing edge of the slat perturb it. The impact of the turbulent shear layer induces unsteady eruptions of vorticity along the wing suction side (Figure 15).
18 of 25 American Institute of Aeronautics and Astronautics
Figure 15. Numerical Schlieren, mid-plane view.
VIII.
Space weather phenomena
A global magnetospheric model based upon Magneto Hydro Dynamics (MHD) has been developed in collaboration with the Centre for mathematical Plasma-Astrophysics of the KU Leuven university.22, 49, 50 The method is based on a B0 + B1 splitting technique, in which the total magnetic field B is given by: B = B0 (planetary dipole magnetic field) + B1 (variable magnetic field).
(5)
1 r 3 (3
(m · nr ) nr − m), with r being the distance from the origin, Herein, the planetary dipole reads B0 (r) = m the dipole moment vector, nr the unit vector in the direction of the position vector. Besides injecting the definition of B into the original system of MHD equations, a Hyperbolic Divergence Cleaning (HDC) approach is applied in order to enforce the solenoidality condition, i.e., ∇ · B = 0. The resulting modified system reads: ∂UHDC−M F S + ∇·FHDC−M F S + ∇·FHDC−Rest = 0 (6) ∂t where the conservative variables UHDC−M F S , the flux vectors FHDC−M F S (completely independent from B0 ) and FHDC−Rest are defined as: UHDC−M F S
=
ρ ρv B E1 = E − B1 · B0 − B02 /2 Φ
,
FHDC−Rest
FHDC−M F S
= F(UHDC−M F S ) =
=
0 (B0 · B1 )¯I − (B0 B1 + B1 B0 ) vB0 − B0 v (B0 · B1 )v − (v · B1 )B0 0
ρv ρvv + (p + B12 /2)¯I − B1 B1 vB1 − B1 v + ¯IΦ (E1 + p + B12 /2)v − (v · B1 )B1 2 Vref B1
This model has been combined with a standard second-order FV discretization relying upon a Lax-Friedrichs scheme with tunable dissipation, in combination with fully implicit one- (for steady cases) and three-point (for unsteady cases) Backward Euler time stepping, as extensively explained in.22, 50 The results of two 3D steady simulations of the interaction between the Solar wind and the Earth’s magnetosphere are presented in Fig. 16(b), one using HDC and the other one using the conventional approach based on the Powell’s 19 of 25 American Institute of Aeronautics and Astronautics
source term. Both cases were run on the shock-adapted mesh shown in Fig. 16(a). The superior accuracy of HDC is clearly visible in Fig. 16(c), where ∇ · B appears to be close to machine accuracy. A 3D view of the overall flowfield surrounding the Earth in terms of density variations, B field lines and contours is shown in Fig. 16(d). More challenging unsteady simulations accepting as input real ACE satellite data are currently being performed and will be discussed in the final paper. The same MHD code here discussed is currently being ported to Graphical Processor Units (GPU) using CUDA as programming language. Preliminary results show a speed up close to 10X when running with Forward Euler, while the extension to implicit time stepping is ongoing.
(a) Second-level adapted grid (2,013,946 tetrahedral cells).
(b) Pressure variations over B field lines in xz-plane: HDC (left) vs. Powell’s source term approaches.
(c) Divergence of B field: HDC vs. Powell’s source term.
IX.
(d) Density variations over B field lines and contours on the z=-20 plane.
High-order methods for conservation laws
Various high-order (i.e., > 2nd order) space discretizations have been developed within COOLFluiD, including FEM,4 RDS,16, 51, 52 SFV and SFD.17, 18 We focus here on describing some recent developments in RDS.
20 of 25 American Institute of Aeronautics and Astronautics
A.
Residual Distribution method
We consider the steady state solution of a conservation law in a two dimensional spatial domain Ω: ∂u + ∇F = 0 ∀(x, y) ∈ Ω, ∀t ≥ 0 , where ∂t F(u) = (f1 (u), f2 (u)) is the flux vector .
(7)
The triangulation of the Ω is denotedy by Th,k , whereby each triangular element T of Th,k consists of a Lagrangian Pk finite element equipped with N = (k+1)(k+2) degrees of freedom as shown in Fig. 16 for 2 P1 , P2 and P3 elements. The equation 7 is discretized by the Residual Distribution Method, extended to high-order isoparametric elements. For any triangulation T h,k the cell residual for a given triangle is defined
Ts
Ts
P1
P2
k=1 N=3
P3
k=3 N=10
k=2 N=6
Figure 16. P1 , P2 and P3 Lagrangian elements with P1 sub-triangulation
as the integral over the triangle of the spatial operator: Z I φT = ∇F(uh,k ) dΩ = T
F(uh,k ) · n dl
∂T
We can also construct the residual over each P1 sub-element of T h,k I Ts φ = F(uh,k ) · n dl ∂Ts
The basic idea of the residual distribution method consists of distributing fractions of this cell residual to the three nodes of the element T . The fraction sent to node i ∈ T is denoted φi and we require X φi = φ T i∈T
This requirement can be written in terms of distribution coefficients βi which for consistency sum to unity X
βi = 1,
βi =
i∈T
φi φT
P After assembling all contributions in the nodes, the nodal equation for node i reads T ∈Di φi = 0, with Di the set of all triangles that share node i. Since the distribution is restricted to the nodes of the triangle itself, the stencil of the scheme remains compact. This equation can be solved by embedding in a pseudo–time iteration, i.e. by finding a steady state solution of |Si |
X dui + φTi = 0, dτ T ∈Di
where |Si | is the area of median dual cell.
21 of 25 American Institute of Aeronautics and Astronautics
1.
Residual distribution on curvilinear elements
We consider finite elements with piecewise quadratic or cubic geometry. The boundary ∂Ts is described by an isoparametric transformation betwen reference and physical space.52 For every face on ∂Ts , this transformation is a function of only one parameter ζ: x = x(ζ),
y = y(ζ)
(8)
We integrate the numerical fluxes using Gauss quadrature: 3 points per face f of the sub–element for P2 and 5 points for P3 : I N X F(uh,k ) · n dl ' wq [F(uq ) · nq ]Jq , f
q=1
q Jq = x˙ 2 (ζq ) + y˙ 2 (ζq )
(9)
where Jq is the Jacobian of transformation (8) at quadrature point q. We use the same transformation to compute the normal nq . 2.
Subsonic flow around NACA0012 airfoil
We use the example of subsonic flow (M ain = 0.5, αin = 2◦ ) around a NACA0012 airfoil to demonstrate the importance of using curvature of the boundaries. The characteristics of the mesh used for each particular element type are listed in table 2. As presented in Fig. 17 , the computed pressure coefficient tends to Elements P1 P1 P1 P2 P2 P2 P3 P3
NDOF 4 931 4 931 4 931 11 016
NE 9 652 2 413 2 413 2 413
NEA 176 88 88 88
Table 2. Meshes for NACA0012 test case. NE : number of elements; NEA : number of element faces on the airfoil
exhibit oscillations in the case of P1 P2 (linear geometry approximation, quadratic solution interpolation) elements. These oscillations are not present with P2 P2 and P3 P3 elements. Moreover, proper modeling of the wall geometry significantly decreases generation of unphysical entropy around the airfoil, as shown in Fig. 18.
Figure 17. NACA0012 - pressure coefficient (top) and detail of Cp close to the leading edge on the suction side of the airfoil (bottom).
22 of 25 American Institute of Aeronautics and Astronautics
Figure 18. Naca0012 - deviation from inlet entropy on the wall of the airfoil. Top row: comparison of solution on elements with no boundary curvature (left) and curved elements (right). Bottom row: detail of solution obtained by P2 P2 and P3 P3 scheme.
X.
Conclusions
This paper has presented a partial but comprehensive review of the capabilities of COOLFluiD, an extremely flexible HPC framework designed for scientific/engineering simulations, particularly (but not exclusively) in the CFD field. A broad range of applications has been discussed, ranging from high-speed chemically reacting flows, to radiation, aeroacoustics, turbulence, MHD, etc. In COOLFluiD not only multiple physical models, but also multiple numerical discretizations with different data structures (e.g., cell-centered FV, FEM, RDS, higher order methods) coexist. This demostrates its suitability to offer an ideal research environment for developing more advanced models and algorithms in order to tackle increasingly complex multi-disciplinary problems.
Acknowledgments The authors wish to express their gratitude to all former contributors to the project, in particular to Dr. Tiago Quintino (co-initiator of the project), Dr. Dries Kimpe and Dr. Thomas Wuilbaut, who all crucially contributed to the early development and ultimate success of the project. We also express our deep acknowledgement to Prof. Herman Deconinck, Prof. Patrick Rambaud (from VKI) and Prof. Stefaan Poedts (from KU Leuven) for their constant support during all those years.
23 of 25 American Institute of Aeronautics and Astronautics
References 1 Lani, A., Quintino, T., Kimpe, D., Deconinck, H., Vandewalle, S., and Poedts, S., “The COOLFluiD Framework: Design Solutions for High-Performance Object Oriented Scientific Computing Software,” Computational Science ICCS 2005 , edited by P. M. A. S. V. S. Sunderan, G. D. van Albada and J. J. Dongarra, Vol. 1 of LNCS 3514 , Emory University, Springer, Atlanta, GA, USA, May 2005, pp. 281–286. 2 Lani, A., An Object Oriented and high performance platform for aerothermodynamics simulation, Phd thesis, Universit´ e Libre de Bruxelles, 2008. 3 Quintino, T., A Component Environment for High-Performance Scientific Computing. Design and Implementation, Phd thesis, Katholieke Universiteit Leuven, 2008. 4 Wuilbaut, T., Algorithmic Developments for a Multiphysics Framework , Ph.D. thesis, Universit´ e Libre de Bruxelles, 2008. 5 Lani, A., Quintino, T., Kimpe, D., Deconinck, H., Vandewalle, S., and Poedts, S., “Reusable Object-Oriented Solutions for Numerical Simulation of PDEs in a High Performance Environment,” Scientific Programming. Special Edition on POOSC 2005 , Vol. 14, No. 2, 2006, pp. 111–139. 6 Lani, A. and Quintino, T., “Design techniques for high performance multi-physics simulations,” Lecture Series on High performance computing of industrial flows, VKI, Rhode Saint Genese, Belgium, May 2009. 7 Lani, A. and Deconinck, H., “Typesafe and Size-Deducing Fast Expression Templates for Small Arrays,” POOSC’06 Workshop Notes, Von Karman Institute, Nantes, Jul 2006. 8 Quintino, T. and Lani, A., “Method-Command-Strategy Pattern: A Multi-Component Solution for High-Performance Scientific Computing,” POOSC’05 Workshop Notes, Von Karman Institute, Glasgow, Jul 2005. 9 Alexandrescu, A., Modern C++ Design, C++ In-Depth Series, Addison-Wesley, 2001. 10 Gamma, E., Helm, R., Johnson, R., and Vlissides, J., Design Patterns - Elements of Reusable Object-Oriented Software, Addison-Wesley Professional Computing Series, Addison-Wesley, 1994. 11 Laboratory, A. N., “PETSc: Portable, Extensible Toolkit for Scientific Computation,” 2007. 12 Karypis, G., “ParMETIS: Parallel Mesh Partitioning,” . 13 Barth, T., “Aspects of unstructured grids and finite volume solvers fro the Euler and Navier-Stokes equations,” 25th Computational Fluid Dynamics Lecture Series, Von Karman Institute, March 1994. 14 Leer, B. V., “Towards the ultimate conservation difference scheme. IV. A new approach to numercial convection.” Journal of Computational Physics, , No. 14, 1979, pp. 361–370. 15 Kapa-Koloszar, L., Numerical prediction of noise production and propagation, Ph.D. thesis, Universite Libre de Bruxelles, 2011. 16 Villedie, N., High order discretisation by Residual Distribution Schemes, Phd thesis, Universit´ e Libre de Bruxelles, 2009. 17 den Abeele, K. V., Development of high-order accurate schemes for unstructured grids, Phd thesis, Vrije Universiteit Brussel, 2009. 18 Parsani, M., Development of an efficient Navier-Stokes/LES solver on unstructured grids for high-order accurate schemes, Phd thesis, Vrije Universiteit Brussel, 2010. 19 Maciel, P., A Numerical Framework for Electrochemistry - Application to the study of gas-evolving electrochemical reactions, Phd thesis, Vrije Universiteit Brussel, 2011. 20 Panesi, M., Physical models for nonequilibrium plasma flow simulations at high speed re-entry conditions, Ph.D. thesis, Universit´ a di Pisa, Von Karman Institute of Fluid Dynamics, 2008. 21 Sartori, E., Numerical simulation of chemical reactive flows in Inductively Coupled Plasma Windtunnels, Master’s thesis, University of Padua, Italy, 2009. 22 Yalim, M. S., An Artificial Compressibility Analogy Approach for Compressible Ideal MHD: Application to Space Weather Simulation, Ph.D. thesis, Universit´ e Libre de Bruxelles, 2008. 23 Sanna, A., Design and Implementation of a Computational Environment for High Performance Scientific Computing, Phd thesis, Universit´ a degli Studi di Cagliari, 2012. 24 Lani, A., Molnar, J., Abeele, D. V., Rini, P., Magin, T., and Degrez, G., Numerical study of elemental demixing in atmospheric entry flow regimes near local thermodynamic equilibrium, edited by E. O. P. Wesseling and Periaux, ECCOMAS 2006, TU Delft, Egmond Aan Zee (Netherlands), Sep 2006. 25 Degrez, G., Lani, A., Panesi, M., and al., “Modelling of high-enthalpy, high-Mach number flows,” J. Phys. D: App. Phys., Vol. 41, 2009. 26 Panesi, M., Lani, A., and Chazot, O., “Reduced Kinetic Mechanism for CFD Applications,” AIAA-2009-3920 , 2009. 27 Munafo, A., Panesi, M., Jaffe, R., Lani, A., and Magin, T., “Vibrational State to State Kinetics in Expanding and Compressing Nitrogen Flows,” AIAA-2010-4335 , 2010. 28 Magin, T., “A model for Inductive Plasma Wind Tunnels,” Ph.d. thesis, von Karman Institute for Fluid Dynamics, St.-Genesius-Rode, Belgium, June 2004. 29 Magin, T. and Degrez, G., “Transport Algorithms for Partially Ionized and Unmagnetized Plasmas,” Journal of Computational Physics, Vol. 198, 2004, pp. 424. 30 Knight, D., Longo, J., Drikakis, D., Gaitonde, D., Lani, A., and al., “Assessment of CFD Capability for Prediction of Hypersonic Shock Interactions,” Prog. Aerosp. Sci., Vol. 48-49, 2012, pp. 8–26. 31 Lani, A. and Deconick, H., “Conservative Residual Distribution Method For Hypersonic Flows In Thermochemical Nonequilibrium,” 47th AIAA Aerospace Science Meeting and Exhibit, Orlando (FL), Jan 2009. 32 Lani, A., Mena, J. G., and Deconick, H., “Residual Distribution Method for Symmetrized Systems in Thermochemical Nonequilibrium,” 20th AIAA CFD Conference, Honolulu (Hawaii), Jun 2011.
24 of 25 American Institute of Aeronautics and Astronautics
33 .Lani, A., Panesi, M., and Deconinck, H., “Conservative Residual Distribution Method For Viscous Double Cone Flows In Thermochemical Nonequilibrium,” Commun. Comput. Phys., Vol. 13, 2012, pp. 479–501. 34 Bensassi, K., Lani, A., Rambaud, P., and Chazot, O., “Numerical Simulation of Hypersonic Flow in VKI-Longshot Contoured Nozzle,” 40th Fluid Dynamics Conference, Chicago (Illinois), Jun 2010. 35 Bensassi, K., Lani, A., Lico, A., Rambaud, P., and Chazot, O., “Unsteady Simulation of Hypersonic Flow over a Heat Probe in Ground Test Conditions,” 7th ESA Symposium on Aerothermodynamics for Space Vehicles, 2011. 36 Bensassi, K., Lani, A., Chazot, O., and Rambaud, P., “Assessment and Validation of the Rebuilding process of Test conditions in VKI-Longshot Hypersonic facility,” AIAA 2011-3886 , 2011. 37 Bensassi, K., Lani, A., Chazot, O., and Rambaud, P., “Arbitrary Lagrangian Eulerian Simulation of a Moving Piston in Hypersonic Ground Test Facility,” New Orleans (Louisiana), Jun 2012. 38 Lani, A., Sanna, A., Villedieu, N., and Panesi, M., “COOLFluiD: an open computational platform for aerothermodynamics and flow-radiation coupling,” Radiation of High Temperature Gas (RHTG), Barcelona, Oct 2012. 39 Tam, C. K. W., “Computational Aeroacoustics: An Overview of Computational Challanges and Applications,” Int. J. of Comp. Fluid Dynamics, Vol. 18, 2004. 40 Colonius, T. and Lele, S. K., “Computational aeroacoustics: progress on nonlinear problems of sound generation,” Progress in Aerospace Science, Vol. 40, 2004. 41 Bailly, C. and Juve ´, D., “Numerical Solution of Acoustic Propagation Problems Linearized Euler’s Equations,” 6th AIAA/CEAS Aeroacoustic Conference, 2000. 42 Tam, C. K. W., “Computational Aeroacoustics: Issues and Methods,” AIAA Journal, Vol. 33, 1995. 43 Koloszar, L., Villedieu, N., Quintino, T., Anthoine, J., and Rambaud, P., “Application of Residual Distribution Method for Acoustic Wave Propagation,” AIAA 2009-3116 , 2009. 44 Koloszar, L., Villedieu, N., Quintino, T., Deconinck, H., Anthoine, J., and Rambaud, P., “Residual Distribution Method for Aeroacoustic,” AIAA Journal, 2011. 45 Koloszar, L., Villedieu, N., Deconinck, H., Anthoine, J., and Rambaud, P., “Extension of non-reflective outlet boundary condition for vortical disturbances,” AIAA-2011-2798 , 2011. 46 “ICASE/LaRC Workshop on Benchmark Problems in Computational Aeroacoustics (CAA),” NASA CP 3300 , 1995. 47 Lockhard, D. P. and Choudhari, M., “Noise radiation from a leading-edge slat,” AIAA -2012-2100 , 2012. 48 Terracol, M., Manoha, E., and Lemoine, B., “Investigation of the unsteady flow and noise sources generation in a slat cove: hybrid zonal RANS/LES simulation and dedicated experiment,” AIAA -2011-3203 , 2011. 49 Yalim, M. S., Abeele, D. V., and Lani, A., “Simulation of field-aligned ideal MHD flows around perfectly conducting cylinders using an artificial compressibility approach,” Proc. of the 11th International Conference on Hyperbolic Problems, Ecole Normale Sup´ erieure, Springer-Verlag, Lyon, France, July 17-21 2006, pp. 1085–1092. 50 Yalim, M. S., Abeele, D. V., Lani, A., Quintino, T., and Deconinck, H., “A Finite Volume Implicit Time Integration Method for Solving the Equations of Ideal Magnetohydrodynamics for the Hyperbolic Divergence Cleaning Approach,” J. Comput. Phys., Vol. 230, No. 15, July 2011, pp. 6136–6154. 51 Villedieu, N., Quintino, T., Ricchiuto, M., and Deconinck, H., “3rd Order Residual Distribution schemes for the NavierStokes equations,” J. Comput. Phys., Vol. 230, No. 11, 2011, pp. 4301–4315. 52 Vymazal, M., Villedieu, N., Quintino, T., and Deconinck, H., “High-order upwind residual distribution schemes on isoparametric curved elements,” J. Comput. Phys., Vol. 230, No. 4, 2010, pp. 890–906.
25 of 25 American Institute of Aeronautics and Astronautics
