R HPC enabling of OpenFOAM for CFD applications HPC simulation of volcanic ash plumes and application of OpenFOAM to CFD volcanological problems 06-08 April 2016, Casalecchio di Reno, BOLOGNA. Matteo Cerminara –
[email protected] Tomaso Esposti Ongaro –
[email protected] Mattia de’ Michieli Vitturi –
[email protected] Istituto Nazionale di Geofisica e Vulcanologia Istituto Nazionale di Oceanografia e Geofisica Sperimentale
Effusive and explosive eruptions: The example of Etna volcano effusive (lava flows)
(lava fountains)
explosive (ash plumes)
(2006)
(2014)
(2015)
Explosivity is mostly controlled by multiphase processes in the magma (melt+gas+crystals): • Gas phase transitions (gas exsolution, bubble nucleation and
expansion). • Non-linear magma rheology (brittle transition = fragmentation). Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
2 / 44
Challenges in magma ascent modeling with OpenFOAM
Effusive regime • Introduce multicomponent physics and phase transitions • Bubble nucleation • Manage phase separation and degassing
Explosive regime • Wave propagation • Fragmentation conditions • Manage different domains (below/above fragmentation)
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
3 / 44
TwoPhaseChangeEulerFoam A new multiphase multicomponent model with phase change has been developed. Each phase is the mixture of several components and exsolution/evaporation laws can be defined for each component.
Test 1 (decompression experiment): • validation through comparisons with decompression experiments performed at LMU, Munich. • silicon oil chosen as analogue for magmatic melt, and saturated with Argon at 10MPA • slow decompression to atmospheric conditions
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
4 / 44
TwoPhaseChangeEulerFoam A new multiphase multicomponent model with phase change has been developed. Each phase is the mixture of several components and exsolution/evaporation laws can be defined for each component.
Test 1 (decompression experiment): • validation through comparisons with decompression experiments performed at LMU, Munich. • silicon oil chosen as analogue for magmatic melt, and saturated with Argon at 10MPA • slow decompression to atmospheric conditions
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
5 / 44
TwoPhaseChangeEulerFoam A new multiphase multicomponent model with phase change has been developed. Each phase is the mixture of several components and exsolution/evaporation laws can be defined for each component.
Test 2 (gas-driven effusive eruption): • rise of hot and high-viscosity magmatic mixture: • liquid phase: melt, dissolved
H2O, dissolved CO2; • gas phase: exsolved H2O,
exsolved CO2, air;
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
6 / 44
Explosive eruptions: Volcanic plumes Weak Plume Strong Plume
(Chait´en, 2008) (Etna, 2011) Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
7 / 44
Multiscale dynamics Spatial and temporal scales • Mass flow rate: Q ∼ UD 2 • Reynolds number: Re =
UD µ
∼ 109
• Typical Large-Eddy Scale: L ∼ (Str)D • Kolmogorov’ length: η ∼ L−3/4 ∼ L × 10−7 • Integral length scale (plume height): H ∼ F 1/4 S −3/4
(F = g 0 UD 2 = g 0 Q; S = −g /ρ dρ dz ) q Tv 2 • Integral time scale: τ ∼ (1+n)Γg ∼ 1 − 5 × 10 s
Number of degrees of freedom • Minimum grid size: ∆x ∼ L • Minimum number of cells: N ∼ (H/L)3 ∼ Q −1/4 • Minimum time-step ∆t ∼ ∆x/U • Ntot (Weak) ∼ 1014 ; Ntot (Strong) ∼ 1011 Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
8 / 44
Challenges in volcanic plume modeling with OpenFOAM
Multiphase flow • Describe non-equilibrium gas–particle dynamics • Manage compressibility, turbulence, heat exchange • Model subgrid turbulence
Numerical solution • Time-step constrained by vent conditions • Compressible turbulence needs appropriate discretization schemes • Number of cells is necessarily large! Effective parallelism.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
9 / 44
Model-related uncertainty Weak Plume
Strong Plume
Does uncertainty blur accuracy? • A ”blind” intercomparison test (4 models) • Model-related uncertainty is still much larger than • Error associated with grid size • Error associated with SGS model • Measurement uncertainty is within the model error Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
10 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
11 / 44
Eulerian fields A new model ASHEE (Ash Equilibrium Eulerian) has been developed taking advantage of the OpenFOAM infrastructure • it models a mixture of I gas species and J solid particle species, each
with diameter dj • the ith gas species is characterized by the following fields in each
point (x, t) • bulk density ρi • velocity ug • temperature Tg
• the jth solid species is characterized by the following fields in each
point (x, t) • bulk density ρj • velocity uj • temperature Tj
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
12 / 44
Mass averaged Eulerian fields The conservation equations for mass, momentum and energy are written focusing on the mass averaged mixture properties:
Mass averaged field ψm Firstly we define the mixture density as ρm = the mass fractions of each phase are defined:
PI
i=1
ρi +
PJ
j=1
ρj , so that
• yi = ρi /ρm • yj = ρj /ρm .
Thus, given a generic field ψ(x, t) for the ith gas species (ψi ) or for the jth solid species (ψj ), we define ψm =
I X i=1
yi ψi +
J X
yj ψ j
j=1
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
13 / 44
Physical configuration The Eulerian model in “mixture” formulation ∂t ρm + ∇ · (ρm um ) =
X
Sj ;
j∈J
∂t (ρm yi ) + ∇ · (ρm ug yi ) = 0,
i ∈I;
∂t (ρm yj ) + ∇ · (ρm uj yj ) = Sj ,
j ∈ J;
∂t (ρm um ) + ∇ · (ρm um ⊗ um + ρm Tr ) = X = −∇p + ∇ · T + ρm g + Sj uj ; j∈J
∂t (ρm hm ) + ∇ · [ρm hm (um + vh )] + + ∂t (ρm Km ) + −∇ · [ρm Km (um + vK )] = = ∂t p + ∇ · (T · ug − q) + ρm (g · um ) +
X
Sj (hj + Kj ) .
j∈J
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
14 / 44
Physical configuration
• The “mixture” formulation is useful in two-way coupled multiphase
systems, where the mass of the dispersed phase has a non-negligible effect on the dynamics. • The problem is formulated in the dispersed regime s . 10−3 ,
collisions between particles are disregarded • It focuses the mathematical problem on the mass averaged fields:
ρm , um , hm (improving stability). • All the effects due to the kinematic decoupling are confined in the
terms Tr (yj , vj ) , vh (yj , vj , hj , hm ) , vK (yj , vj , Kj , Km ) , keeping into account the effect of the relative velocity vj = uj − ug • No explicit dependence on the drag functional expression is
necessary at this level
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
15 / 44
Equilibrium Eulerian model
• if particles are perfectly coupled both thermally (Tj = Tg = T ) and
kinematically (uj = ug = u) we recover the dusty gas model, where all the decoupling terms are zero • in ASHEE we adopt the equilibrium-Eulerian model, where the
system is thermally perfectly coupled while the kinematic decoupling is approximated via an asymptotic expansion relative to the Stokes time τj : uj = ug + wj − τj (ag + wj · ∇ug ) + O(τj ) where wj = τj g is the settling velocity and ag is the gas phase acceleration
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
16 / 44
Equilibrium Eulerian model
• The contribution of the particle inertia can be accurately taken into
account by using standard Navier-Stokes numerical algorithm, without needing to implicitly solve the drag term • Particle decoupling and preferential concentration well modeled up
to St . 0.2, keeping the advantages of the dusty gas model • Total number of equation highly reduced for a polydispersed mixture
(4J PDEs less): • Eulerian: I + 3 + 5J • Equilibrium-Eulerian: I + 3 + J.
• Allows to solve efficiently the multiphase dynamics at geophysical
scale for particles of size up to ' 1 mm.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
17 / 44
Paper on the ASHEE model
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
18 / 44
Numerical configuration
We implemented the following subgrid-scale LES models for the subgrid terms of the compressible equilibrium-Eulerian model: • Compressible Smagorinsky (static and dynamic) • Turbulent Kinetic Energy model (static and dynamic) • WALE model (static and dynamic)
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
19 / 44
Numerical configuration
We modified the compressible monophase PISO-PIMPLE algorithm: • predictor for the mixture density ρm • PIMPLE loop • solve for the mass fractions yi,j • predictor for the mixture velocity um • solve for the enthalpy hm , fixing the compressibility • PISO loop • solve the decoupling • solve for pressure p • correct velocity and fluxes • solve LES models
• correct mixture density
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
20 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
21 / 44
DHIT Decaying Homogeneous and Isotropic Turbulence (2563 cells) The solver is able to simulate accurately the turbulence. 10
-2
10-3 10
-4
E(k)
10-5 10-6 10
-7
10-8 10
DNS Bernardini & Pirozzoli OpenFoam
-9
10-10
100
101
102
k
Figure:
Test case validation: comparison with an eight order DNS after one large-eddy turnover time (10000 time steps).
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
22 / 44
1200
1,10 1,00 0,90 0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00
1000 800 Speed up
Efficiency
Scalability
600 400 200
64 Fer mi PLX
128
256 # cores
512
1024
0 64
128
Fermi
256
512
1024
# cores
Ideal PLX
Figure: Scalability test on PLX and FERMI environments (with little output work). Collaboration between us and Paride Dagna. In order to fix ideas, the solver reach a velocity in the range 1÷10 Mcells/s on 1024 cores (multiphase÷monophase) .
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
23 / 44
SGS LES models
energy spectrum E(k)
0.001
0.0001
1e-05 3
[noM] 2563 [noM] 323 [sma] 32 [oneEqEddy] 3233 [wale] 32
1e-06 1
10 wavenumber k
Figure: DHIT using static SGS LES models in a box with 323 cells
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
24 / 44
SGS LES models
energy spectrum E(k)
0.001
0.0001
1e-05
[noM] 2563 [dynSma] 6433 [moin] 32 [dynSma] 323 [dynOneEqEddy] 323 [dynWale] 323 [cubic] [dynOneEqEddy] 323
1e-06 1
10 wavenumber k
Figure: DHIT using dynamic SGS LES models in a box with 323 cells
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
25 / 44
Kinematic decoupling Slice of the turbulent box at t/τe ' 2.2. The two panels represent respectively a logarithmic color map of yj=2 (Stmax = 0.5) and of |ag |
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
26 / 44
Kinematic decoupling 1 3
τj τξ 〈P〉j (LES);
τj τη 〈P〉j
(DNS)
0.1
[noM], 256 3 [dynWale], 32 1.52*St2 Rani 2003 fit of Rani 2003
0.01
0.001
0.0001
1e-05 0.01 Stξ (LES);
0.1 Stη (DNS)
1
Figure: Evolution of the degree of preferential concentration with Stξ (LES) or Stη (DNS). We obtain a good agreement between equilibriumEulerian LES/DNS and Lagrangian DNS simulations.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
27 / 44
Advection schemes
Figure: Advection test case from Holzmann-cfd solved with ASHEE
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
28 / 44
Advection schemes 1.2
1
0.8
tracer
0.6
upwind linearUpwind linear limitedLinear limitedLimitedLinear vanLeer limitedVanLeer cubic limitedCubic UMIST SFCD QUICK filteredLinear
0.4
0.2
0
-0.2 -0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
time
Figure: Tracer mass fraction along the cavity section. Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
29 / 44
Advection schemes 10
-3
10-4
[noM], 2563
E(k)
[dynWale], 323 [dynWale], [MUSCL], 323 [dynWale], [UMIST], 323 [dynWale], [vanLeer], 323 [dynWale], [rk4blended], 323 10-5
10-6
[dynWale], [filtlin0200], 323
1
10 k
Figure:
Performances of OpenFOAM schemes in DHIT LES
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
30 / 44
Other benchmarks 1
10
analytic solution rhoCentralFoam ASHEE
0.9
Ra = 1e6, Nu = 8.800
0.8 Ra = 1e5, Nu = 4.519
0.7
1 0.9
0.6
Nu
ρ
0.8 0.7
0.5
0.6 ρ
Ra = 1e4, Nu = 2.243
0.5
0.4
0.4
0.3
0.3 0.2
0.2
0.1
-2
-1.5
-1
-0.5
0
0.5
1
1.5
Ra = 1e3, Nu = 1.118
2
x/ct
0.1
-2
-1.5
-1
-0.5
0 x/ct
Sod’s shock tube
0.5
1
1.5
2
1 10
100 t [s]
1000
Natural convection
Experimental plume Mixing Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
31 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
32 / 44
Plinian eruption We have used ASHEE to simulate volcanic eruptions (scale ≈ 100 km) from the vent (mass eruption rate up to Mton/s) to the atmosphere
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
33 / 44
Volcanic plumes
We are participating to an international benchmark initiative involving two numerical simulations:
weakPlume
strongPlume
• duration: 0.2 hours
• duration: 2.5 hours
• Mass flow rate: 1.5 ∗ 106 kg/s
• Mass flow rate: 1.5 ∗ 109 kg/s
• Exit velocity: 135 m/s
• Exit velocity: 275 m/s
• Exit temperature: 1273 K
• Exit temperature: 1053 K
• Exit gas fraction: 3 wt% • Grain size distribution:
• Exit gas fraction: 5 wt% • Grain size distribution:
• coarse: 1 mm • fine: 62.5 µm
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
• coarse: 0.5 mm • fine: 15.6 µm
34 / 44
Effects of resolution and SGS model Time and horizontal average of the velocity field, with 8, 16 and 32 cells in a vent diameter; with and without decoupling model (weakPlume) 14
[dynWale], high res., [eqEu] [dynWale], mid. res., [eqEu] [dynWale], low res., [eqEu] [dynWale], low res., [dusty]
12
z [km]
10 8 6 4 2 0 0
20
40
60
80
100
120
140
U [m/s] Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
35 / 44
Effects of resolution Time and horizontal average of the velocity field, with 8, 16 and 32 cells in a vent diameter (strongPlume) 45
[dynWale], high res., [eqEu] [dynWale], mid. res., [eqEu] [dynWale], low res., [eqEu]
40 35
z [km]
30 25 20 15 10 5 0 0
50
100
150
200
250
300
U [m/s] Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
36 / 44
Effects of SGS model Time and horizontal average of the velocity field, with 16 and 32 cells in a vent diameter; with different SGS models (strongPlume) 45
[dynWale], high res., [eqEu] [noM], high res., [eqEu] [dynWale], mid. res., [eqEu] [moin], mid. res., [eqEu] [oneEqEddy], mid. res., [eqEu]
40 35
z [km]
30 25 20 15 10 5 0 0
50
100
150
200
250
300
U [m/s] Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
37 / 44
Volcanic infrasound Compressibility and turbulence generate infrasound
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
38 / 44
Observations & simulations Observation of 8 March 2005 Mount St. Helens eruption (Matoza et al. Simulation with ASHEE of the 2009) strongPlume eruption f [Hz] 0.01
0.1
1 70 Ep
100
60
40 0.1
0.01
1.2 1 0.8 0.6 0.4 0.2 0 -0.2 -0.4 -0.6
Ep(Str) [dB re 20 µPa]
50
-11/4
1
p [kPa]
Ep(Str) [Pa]
10
30
0
0.001 0.01
100
200
300
400 t [s]
500
600
700
800
20
0.1
1
10
Str
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
39 / 44
Summary of the second part
• We have developed the compressible version of the
equilibrium-Eulerian model (ASHEE model) • We have implemented it into the OpenFOAM infrastructure • the solver has been tested up to 1024 cores. It shows a reasonable
efficiency (> 60%) on the Cineca Fermi infrastructure • The numerical scheme has been tested and chosen to maximize
accuracy and stability of dynamic LES simulations • A number of different benchmarks have been performed satisfactorily • The new solver is able to accurately and efficiently capture
clustering, preferential concentration and settling up to 1 mm particles • Applications to volcanological scale have been performed, comparing
results with other models and with observations
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
40 / 44
Volcanic hazard assessment: an HPC/HTC challenge
Complex physics, multiscale dynamics • Non-newtonian (non-linear) rheology • Multiphase flows • Broad range of spatial/temporal scales
Uncertainty on initial/boundary conditions • Partial knowledge of the initial state • Catastrophic dynamics • Difficulty of measurements • Repeatability issues
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
41 / 44
Future perspectives
Magma ascent modeling • Implementation of fragmentation conditions • More complex geometries • Fluid-structure interaction
Volcanic plume modeling • Add an arbitrary wind field • Add volcano topography • Add micro-physics, as water condensation, ash particles shape and
aggregation
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
42 / 44
Future perspective
Numerical issues • Non-reflecting boundary conditions • Improve the advection scheme • Implement a shock capturing correction • Optimize parallel linear algebra in OpenFOAM
Volcanic hazard assessment • Sensitivity and uncertainty analysis (coupled with Dakota). • Coupling with meteorological solvers. • ”Nowcasting” of volcanic plume scenarios.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
43 / 44
Thank You! Contacts • research web-page:
https://sites.google.com/site/matteocerminara • CFD movies:
https://www.youtube.com/user/MatteoCerminara
Acknowledgments: we acknowledge CINECA for the availability of high-performance computing resources and technical support on porting OpenFOAM on HPC architectures in the framework of ISCRA projects: IsB06 VolcFOAM, IsC26 VolcAshP and IsC07 GEOFOAM. Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling
44 / 44
