ANS MC2015 - Joint International Conference on Mathematics and Computation (M&C), Supercomputing in Nuclear Applications (SNA) and the Monte Carlo (MC) Method • Nashville, TN • April 19-23, 2015, on CD-ROM, American Nuclear Society, LaGrange Park, IL (2015)
PETSC-BASED PARALLEL SEMI-IMPLICIT CFD CODE GASFLOW-MPI IN APPLICATION OF HYDROGEN SAFETY ANALYSIS IN CONTAINMENT OF NUCLEAR POWER PLANT Jianjun Xiaoa, John R. Travisb, Peter Royla, Anatoly Svishcheva, Thomas Jordan a and Wolfgang Breitunga a
Institute of Nuclear and Energy Technologies Karlsruhe Institute of Technology P.O. Box 3640, 76021 Karlsruhe, Germany
[email protected];
[email protected];
[email protected];
[email protected];
[email protected] b Engineering and Scientific Software Inc. 3010 Old Pecos Trail, Santa Fe, New Mexico 87505, U.S.A.
[email protected]
ABSTRACT GASFLOW is a CFD software solution used to predict fluid dynamics, heat and mass transfer, chemical kinetics, aerosol transportation and other related phenomena during a postulated severe accident in the containment of nuclear power plant (NPP). The generalized 3-D transient, twophase, compressible Navier-Stokes equations for multi-species are solved in GASFLOW, using a proven semi-implicit pressure-based algorithm of Implicit Continuous Eulerian - Arbitrary Lagrangian-Eulerian (ICE’d-ALE) methodology which is applicable for all flow speeds. GASFLOW has been intensively validated with international experimental benchmarks, and has been widely used in the hydrogen explosion risk analysis involving NPP containments. The simulation results of the GASFLOW code have been widely accepted by the nuclear authorities in several European and Asian countries. GASFLOW was originally designed as a supercomputer serial code and could be only run on vector machines with a single processor. A project was initialized in 2013 in order to parallelize GASFLOW using the paradigms of Message Passing Interface (MPI) and domain decomposition. The data structure and parallel linear solvers in the Portable Extensible Toolkit for Scientific Computing (PETSc) were employed in the GASFLOW parallel version: GASFLOW-MPI. GASFLOW-MPI has been validated using the well accepted benchmarks by the CFD community. The computational time can be dramatically reduced depending on the size of the problem and the high-performance computing (HPC) cluster. GASFLOW parallelization adds tremendous value to large scale containment simulations by enabling high-fidelity models, including more geometric details and more complex physical phenomena that occur during a severe accident, which yield detailed and precise insights. Key Words: high performance computing, computational fluid dynamics, GASFLOW code, PETSc, hydrogen safety analysis, hydrogen risk mitigation, containment, nuclear power plant
1
INTRODUCTION
Hydrogen will be produced by the interaction of molten corium and water during a severe accident in a nuclear power plant [1]. Hydrogen, steam and other species are introduced into the containment through a break located at either the hot or cold leg of the reactor coolant loop, along the surge line, in the pressurizer, or in the primary circuit depressurization system. Burnable hydrogen clouds may form inside the containment. Energetic hydrogen events may threaten the integrity of the containment and lead to radioactivity release to the environment [2].
Jianjun Xiao et al.
In March 2011, hydrogen explosions occurred at the reactor buildings housing Units 1, 3, and 4 of Fukushima Daiichi NPPs causing large releases of harmful radionuclides that contaminated a wide area and prompted the evacuation of some 90,000 people [3]. In March 1979, partial core disruption of a pressurized water reactor (PWR) resulted in a melt down at the Three Mile Island Unit 2 (TMI-2). A milder hydrogen combustion event, which did not breach the containment, also occurred at TMI-2. After the Three Mile Island accident, the USNRC looked to Los Alamos and CFD for containment analysis tools and the HMS code [4-5] was developed for Hydrogen Mitigation Systems within nuclear reactor containments. Later under USDOE funding, HMS developed evolved into the GASFLOW code [6]. In 1995, a long fruitful 10-year collaboration began between LANL and the Forschungszentrum Karlsruhe (FZK, which is now Karlsruhe Institute of Technology - KIT) where the FORTRAN-90 version GASFLOW-II [7-9] came into play. After the collaboration with Los Alamos was not renewed, GASFLOW-III [10-12] was developed at KIT, and now the first publication of the parallel GASFLOW-MPI version is presented. GASFLOW has an extensive verification and validation base [12, 17-29], as presented in Section 4.1. In the past decades, it has been widely used for hydrogen safety analyzes of various nuclear reactor types [12, 31-41], as illustrated in Section 4.2. The simulation results are well accepted by the nuclear authorities in several European and Asian countries. GASFLOW was originally designed as a sequential CFD code, and was executed principally on vector supercomputer machines with only one single processor. For a typical LOCA scenario (10,000 seconds) of PWR containment with around 300,000 cells, it may take roughly 3-4 months for an Intel I7-950 @ 3.07 GHz processor to execute the calculation, which was unacceptable for most of the industrial application users. Obviously, the lack of parallel computing capability has heavily limited the GASFLOW applications in large scale engineering problems. Therefore, a project was initiated in 2013 in order to parallelize GASFLOW using the paradigms of Message Passing Interface (MPI) and domain decomposition. The data structure, parallel linear solvers and preconditioners of Portable Extensible Toolkit for Scientific Computing (PETSc) [13] were employed in the GASFLOW parallel version: GASFLOW-MPI. GASFLOW parallelization adds tremendous value to the numerical simulations of safety analysis in large scale nuclear containments. It enables the creation of large scale and high-fidelity models, such as more complex physics and more geometrical details, which provide accurate and detailed insights into the thermal hydraulics and safety analyses in the containment during a severe accident. Using GASFLOW-MPI to understand the more detailed physical behaviors under severe accidents provides the users confidence in the design of the risk mitigation measures and helps to fulfill the requirements of the nuclear authorities. In this paper, GASFLOW-SEQ represents sequential version of GASFLOW, and GASFLOW-MPI means the GASFLOW parallel version. When only GASFLOW is used, it has a general meaning. 2
GASFLOW: CFD CODE FOR HYDROGEN SAFETY ANALYSIS UNDER SEVERE ACCIDENT
2.1 Basic features GASFLOW is a finite-volume code based on robust computational fluid dynamics numerical techniques that solve the compressible Navier-Stokes equations for 3-D volumes in Cartesian or cylindrical coordinates. The code can model geometrically complex facilities with multiple compartments and internal structures in a computational domain of multiple 3-D blocks of cells. Page 2 of 14
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
GASFLOW has the capabilities of simulating turbulent flow, multi-phase flow, conjugated heat and mass transfer, radiation heat transfer, chemical kinetics, and aerosols. Principally, GASFLOW specializes in the field of containment analysis with many useful unique features, including Fractional Area/Volume Object Representation (FAVOR), a two-phase pre-expansion model, spray model, water film model, recombiner and ignitor models, fan model, sump model, rupture disc model, data reader from lump parameter source codes, criteria for hydrogen safety analysis: flame acceleration (σ criterion) and deflagration to detonation transition (λ criterion). 2.2 Governing Equations The generalized conservation equation d dV b u AdS S dV (1) dt V S V where b is the velocity of the control surface S, u is the velocity of the fluid, and A is the outward normal fractional area vector. SΦ is the source term of Φ. Volume equation d dV b u AdS SV dV (2) dt V S V Mass equation for specie α d dV b u A J A dS S , dV ; (3) dt V S V Where Jα·A is the mass diffusion flux vector. Momentum equation d udV u b u A p τ A Dd A dS g S m dV (4) dt V S V Where Dd·A is the internal structure drag vector. Internal energy equation d p Vh 2o IdV I b u A p u A q A dS S I dV (5) dt V V t S V Where q·A is the Internal energy flux vector, and the effect of phase change can be computed as p Vh 2o Rh 2oTS ,h 2o (6) V t 2.3 ICE’d ALE numerical methodology The ICE’d-ALE numerical methodology is extensively presented by Hirt [15]. This technique is applicable to flows at all speeds meaning it is valid within both sub- and super-sonic flow regimes because it has an implicit formulation similar to that found in the implicit continuousfluid Eulerian (ICE) method [16]. This method is time-split into three distinct phases: Phase A: An explicit Lagrangian phase with some local or point wise implicit usage, where the diffusion terms and source terms are solved; Phase B: An implicit Lagrangian phase where pressure waves are propagated without time-step restrictions; Phase C: An explicit convection (rezone or remapping) phase.
Page 3 of 14
Jianjun Xiao et al.
3
PARALLELIZATION OF GASFLOW USING PETSC LIBRARY FOR MESSAGE PASSING AND DOMAIN DECOMPOSITION
PETSc is one of the most widely used software library for high-performance computational science [13]. It can provide numerical infrastructure for application codes in which the implicit numerical solution of partial differential equations are involved. PETSc features distributed data structures, such as index sets, distributed vectors and distributed matrices in several sparse storage formats, as the fundamental objects. Krylov subspace methods, preconditioners and Newton-like nonlinear methods are implemented in a data structure-neutral manner which provides a uniform interface for application programmers. PETSc can be used for application codes written in FORTRAN, C, C++, Python and MATLAB. The portability of PETSc is achieved through MPI, but the detailed message passing required during the coordination of the computations is handled inside the PETSc library. It helps the application programmers to avoid the difficulties in writing the sophisticated message-passing codes. By using the portable and efficient numerical libraries, the application programmers can concentrate on the physics modeling and accomplish complicated parallel application codes with relatively less effort. Considering that structured grid is used in GASFLOW code, the PETSc data management object, DMDA, for domain decomposition was chosen for the parallelization of GASFLOW. The object contains parallel data layout and communication information, and manages the parallel communication required while working with data stored in regular arrays. Sparse symmetric system is derived from the discretization of the elliptic pressure equation in GASFLOW-MPI. There are various solvers and preconditioners available in PETSc to solve large scale sparse symmetric systems. The CG solver and BJACOBI preconditioner were selected for the solution of the elliptic pressure equation in the current version of GASFLOW-MPI code. Future code development will involve more efficient preconditioners in libraries, such as Hypre or Trilinos, which can further improve the performance of PESTc based GASFLOW-MPI code. 4
TESTING OF GASFLOW-MPI
4.1 Validation matrix of GASFLOW GASFLOW has been verified and validated with many analytical problems largely from text books and since its first development until today has been continuously applied in validation calculations for numerous experiments. Many of the analyzed experiments were performed in the framework of international standard benchmarks and problems. A code assessment manual [12] and a detailed documentation of the relevant analytical and experimental test problems with tables of the test data and the GASFLOW inputs and results comes with the code in a code validation library. Most of the experiments simulate specific sequences and phenomena from severe accidents in nuclear power plants with steam and hydrogen distribution and condensation, catalytic hydrogen/oxygen recombination and hydrogen combustion and thermal and mechanical structure loadings. 4.2 Application cases of GASFLOW-SEQ in nuclear industry Application of GASFLOW for simulating steam/hydrogen distribution and hydrogen combustion in reactor containments first involves the development of 3D models of the reactor containments. Models are developed from technical drawings of the containment building and use either Cartesian or cylindrical meshes. Model developments have been made with our international Page 4 of 14
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
Table I. Relevant experiments of thermal hydraulics in GASFLOW-SEQ validation library Facility Thermal hydraulics HDR Battelle PHEBUS
THAI
MISTRA TOSQAN PANDA SETH CALIST Hydrogen recombination Battelle THAI Hydrogen combustion Battelle SANDIA Bureau of Mines HDR ENACCEF HYMIT
Test
Year
T31.5 (ISP23) E11.2 (ISP29) JX7 HYJET 4A/4B/6A/10A TH1 TH7 TH10 TH10 (ISP47) HM1 HM2 ISP47 ISP47 SARNET-2 9/9bis/21/21bis/25 SARNET-2
1995, 2000 [12,17] 1996, 2004 [12, 18, 19] 1996 [12] 1996 [12] 2002 [21] 2003 [21] 2003 [21, 22] 2006 [12] 2008 2008 [23] 2003 [22], 2010 [24] 2004 [21, 22] 2007 [25] 2007 [26] 2012 [27]
GX4, GX6, GX7 HR3
1996, 1998 [12, 28, 29] 2013
HX12, HX14, HX23 Flame bm_10, bm_20, bm_296 E12.1.2, E12.2.2, E12.3.3 RUN-153, RUN-158 SJTU
1995 [12] 1996 [12] 1996 [12] 1995, 2014 [12] 2014 2014 [30]
industrial partners (Table II) who have detailed knowledge of the building and apply GASFLOW to simulate gas distributions in the building with steam/water/hydrogen sources at specified release locations. 4.3 Testing of GASFLOW-MPI GASFLOW-MPI has been tested using a Standard Test Matrix of problems in five categories. These tests are by now partially completed. (1) Feature or verification tests for the computer science aspects of the code; (2) Functional tests for code algorithms, equations, logic paths, and decision points; (3) Comparisons of computational results with analytical solutions; (4) Comparisons of computational results with experimental data, as shown in Table I; (5) Comparisons of computational results with previous results of large scale industrial applications, as shown in Table II. 4.3.1 Sod’s shock tube A one dimensional model of air was used to test the ability of ICE’d ALE algorithm in GASFLOW-MPI to solve fluid dynamics problems with shock wave behavior [42]. The initial condition is presented in Figure 1. The case was run to a maximum time of 0.0061 seconds. Both first-order upwind and second-order Van Leer schemes in GASFLOW-MPI were tested. Both schemes obtained good agreement when compared with the analytical solutions. The Page 5 of 14
Jianjun Xiao et al.
Table II. Containment models for GASFLOW-SEQ with industrial development users Reactor Year Users HDR Konvoi (GKN-2) EPRTM PWR (Biblis A) VVER-440 PWR (KCB) APR1400 OPR1000 VVER-1000 CNP600 (Qinshan-2) ITER ACPR1000 & EPR PWR PWR BWR4 HTR CAP1400 BWR (Laguna Verde) PWR APR1400
1995/2004 1998/2000 Since 1999 1999 Since 2000 2001 Since 2002 2002 2002 Since 2006 Since 2007 Since 2009 Since 2011 Since 2011 2012 Since 2012 Since 2012 Since 2012 Since 2013 2014
KIT, Germany [12, 17-19] GRS, Germany [31] Areva GmbH, Germany [32] GRS, Germany [33] NUBIKI, Hungary [34] VROM, Netherland KAERI, Korea [35] KAERI, Korea [36] University Wien, Austria [37] Shanghai Jiaotong University, China [38] KIT, Germany [39] China Guangdong Nuclear Power Company (CGNPC), China Nuclear Power Institute of China (NPIC), China North China Electric Power University, China BKW Energy Ltd. , Switzerland Tsinghua University, China [40] State Nuclear Power Technology Company (SNPTC), China ININ, Mexico Nuclear and Radiation Safety Cneter, China Kepco E&C, Korea [41]
computational results of pressure, density and velocity using the first-order upwind numerical scheme are compared with the analytical solutions in Figure 2Figure 4. The effect of mesh resolution was also investigated. It affirms that GASFLOW is capable of accurately capturing the shock wave. 1.0 GASFLOW-MPI (1000 cells) GASFLOW-MPI (750 cells) GASFLOW-MPI (500 cells) GASFLOW-MPI (250 cells) GASFLOW-MPI (100 cells) analytical
pressure (bar)
0.8
0.6
0.4
0.2
0.0 0
1
2
3
4
5
6
7
8
9
X (m)
Figure 1. Initial condition of 1-D shock tube
Figure 2. Pressure profiles at 0.0061 s
Page 6 of 14
10
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
1.0
350 GASFLOW-MPI (1000 cells) GASFLOW-MPI (750 cells) GASFLOW-MPI (500 cells) GASFLOW-MPI (250 cells) GASFLOW-MPI (100 cells) analytical
0.8
GASFLOW-MPI (1000 cells) GASFLOW-MPI (750 cells) GASFLOW-MPI (500 cells) GASFLOW-MPI (250 cells) GASFLOW-MPI (100 cells) analytical
300
0.6
velocity (m/s)
density (kg/m3)
250
0.4
200
150
100
0.2 50
0.0
0
0
1
2
3
4
5
6
7
8
9
X (m)
Figure 3. Density profiles at 0.0061 s
10
0
1
2
3
4
5
6
7
8
9
10
X (m)
Figure 4. Velocity profiles at 0.0061 s
4.3.2 Lid-driven cavity flow The purpose of this test is to verify the ability of the GASFLOW-MPI code to simulate laminar fluid flow with Mach number near to zero in a two-dimensional a lid-driven cavity [43]. Liddriven cavity flow retains a rich fluid flow physics manifested by multiple counter rotating recirculating regions on the corners of the cavity depending on the Reynolds number. A square cavity with a height H = 2 m (Figure 5) has been set up. The top wall moves with the velocity of 1 m/s in X direction. The bottom and side walls are no-slip and stationary. The Reynolds number of the flow is 1000. A non-uniform mesh with 80*80 in X and Y axis has been utilized in the GASFLOW-MPI simulation. The contours of velocity magnitude and stream function were plotted in Figure 6Figure 7. The center-line velocities using the first-order upwind and second-order Van Leer schemes in GASFLOW-MPI were compared with the experimental data, as shown in Figure 8. There exist small discrepancies when first order scheme was used. It indicates that second-order scheme is essential for capturing correct flow features in this lid-driven flow case.
Figure 5. schematic of 2-D lid-driven cavity Figure 6. Contours of velocity magnitude for flow flow in lid-driven cavity
Page 7 of 14
Jianjun Xiao et al.
1.0 W (Ghia, 1982) U (Ghia, 1982) W (2nd Van Leer, GASFLOW-MPI) U (2nd Van Leer, GASFLOW-MPI) W (1st upwind, GASFLOW-MPI) U (1st upwind, GASFLOW-MPI)
0.8
0.6
U / W (m/s)
0.4
0.2 0.0
-0.2
-0.4
-0.6 -1.0
-0.5
0.0
0.5
1.0
Z / X (m)
Figure 7. Plot of streamfunction for flow in liddriven cavity 5
Figure 8. Velocities along the centerlines
APPLICATIONS OF GASFLOW-MPI IN 3-D LARGE SCALE SIMULATIONS
From 2009 to 2012, GASFLOW-SEQ was run on NEC SX-8R at Steinbuch Center for Computing (SCC) of KIT. It used to be the world’s most powerful vector supercomputer. It had the world’s fastest (in 2006) single-clip vector processor with the peak vector performance at 36 GFLOPS. The single-node model (include up to 8 cores) achieved a peak vector performance of 288 GFLOPS. The total memory was 256 GB. Since 2014, we get an ability to utilize our internal GASFLOW dedicated server for the testing of GASFLOW-MPI. The server has Intel Xeon Processor E5-2667 v2 CPU with 3.3 GHz frequency. There are totally 8 nodes with 16 cores on each of them. Each node has 32 GB DDR3-1866 MHz memory. 5.1 Parallel scalability of GASFLOW-MPI: 3-D H2 bubble with complex physics In order to test parallel performance of the GASFLOW-MPI code, a hypothetical 3-D H2 bubble in mixture of air, steam and liquid droplets was simulated. To overweigh the communication cost, most of the features in GASFLOW-MPI, such as the mass and energy diffusions, momentum diffusion, conjugate heat and mass transfer, radiation heat transfer, and second-order Van Leer scheme were switched on. The most time consuming part is solving the sparse matrix derived from the elliptic pressure equation. When the time step is big and the convergence criterion is small, it takes considerable number of iterations until the linear solver is converged. In our testing case, the time step was automatically controlled and the initial time step was set to 0.001 second. The convergence criterion of the linear solver was 1.0e-8. The number of computational cells was 8,000,000. The wall clock times of GASFLOW-SEQ with a single process and GASFLOW-MPI with various numbers of processes were shown in Figure 9. Figure 10 shows the speed-up relative to one process which means the computational time with a single process is used as the base to calculate the speed-up. It seems that very good scaling was obtained. With 128 processes on our GASFLOW-server, a speed-up of 56 was achieved. As expected, the scaling is lower than the ideal one. This is due to the reason that the performance of linear solver is bounded by the memory bandwidth per core. As we know, there are 16 cores on each node on our server. As the blue curve shows in Figure 10, the speed-up slows down with the number of processes increasing from 1 to 16. Apparently, the memory bandwidth becomes Page 8 of 14
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
the bottleneck of the performance when shared by multi cores on the same node. In Figure 11, the total number of cores is fixed at 16. We studied the effect of the number of cores per node. When the number of cores per node is 16, it means only one node is using and the memory bandwidth is shared by 16 cores. A speed-up of 6.8 was obtained. With used cores per node becomes less, the speed-up increases. With 2 cores per node, the speed-up can reach 16 with totally 16 cores. Apparently, under such a situation the memory bandwidth for each core is sufficient and the scaling is not bounded by the memory bandwidth. Figure 12 demonstrates the speed-up relative to one node (16 processes). In our 3-D problem with 8,000,000 cells, a linear scaling was achieved based on the time of 16 processes. 3500
60
GASFLOW-MPI GASFLOW-seq
3000
50
40 2000
Speed-up
Wall clock time (s)
2500
1500
1000
30
20
500
scaling (1-128) scaling (1-16) ideal scaling
10
0 124 8 1216
32
48
64
80
96
112
128
0 1
Number of processes
16
32
48
64
80
96
112
128
Number of processes
Figure 9. Wall clock times
Figure 10. Speed-up relative to one process 10
16
9
GASFLOW-MPI
14
8
Total number of cores = 16
7
12
6
Speed-up
10
speed-up
GASFLOW-MPI Linear scaling
8 6
5 4 3 2
4
1
2
0 16
0 2
4
8
16
32
48
64
80
96
112
128
Number of processes
Number of cores per node
Figure 11. Effect of the number of used cores per node on the speedup
Figure 12. Speed-up relative to one node (16 processes)
5.2 Speed-up of Konvoi type PWR simulation Konvoi type PWR was designed by Siemens AG Germany [31]. The Konvoi Containment is a spherical containment with an outer concrete shell an annular gap of air and a spherical steel shell around a free gas volume of 70,000 m3. The two-phase steam and hydrogen sources were taken from analyses with MELCOR 1.8.3 performed by GRS for these MBLOCA. They cover the in-vessel phase through failure of the core support with total problem time of 35,000 s.
Page 9 of 14
Jianjun Xiao et al.
It was simulated in GASFLOW with a 3-D Cartesian geometry model using 190,125 computational cells with 65 x, 65 y, and 45 z meshes, as shown in Figure 13. Complex physics including conjugated heat and mass transfer, heat conduction, radiation heat transfer, two-phase flow, recombiner, Xenon decay, etc. were simulated in the complicated geometry model. Two cases, with and without radiation heat transfer, were calculated on the vector supercomputer NEC SX-8R using GASFLOW-SEQ in 2011 and on GASFLOW-server using GASFLOW-MPI with 32 processes in 2014. Without radiation model, the ratio of computational time on SX-8R and GASFLOW-server with 32 processes is 6.17. With radiation model, the ratio is 6.45. It seems the performance of linear solver and preconditioner in the radiation model, in which a sparse matrix needs to be solved, have been improved by using PETSc library. It should be noted that SX-8R used to be the fastest vector supercomputer in the worldwide in 2006. Compared to a Intel I7 CPU with frequency of 3.07 GHz, SX-8R is around 4 times faster in our case. It can be roughly estimated that in our Konvoi case GASFLOW-MPI running with 32 processes on our GASFLOW dedicated server could be more than 24 times faster than GASFLOW-SEQ running on a typical PC with Intel I7 CPU @ 3.07 GHz. As shown in Figure 14, the calculation results of GASFLOW-SEQ and GASFLOW-MPI are nearly identical. The dash lines are the results of GASFLOW-MPI, and the solid ones are the predictions of GASFLOW-SEQ. It indicates that GASFLOW-MPI can reproduce the previous results with very good performance speed-up. 5.3 Speed-up of APR1400 simulation The APR1400 is the new Korean commercial PWR with an output of 1400 MW [41]. The objective of the analysis is the evaluation of the mitigation effect of four spray lines in the containment dome that are activated at a certain time in the LBLOCA scenario. The applied water/steam/hydrogen source injects 55 tons of water, 266 tons of steam and 1.28 tons of hydrogen over a time period of 20,000 s. The GASFLOW containment model was developed in a cylindrical mesh with 18 radial, 61 azimuthal, and 41 axial nodes, as shown in Figure 15. The containment model has a free gas volume of 93,724 m3. The total number of cells is 45,018. According to Section 3.2, 10,000-20,000 cells in each computational sub-domain are highly recommended in order to obtain a good scaling. Therefore, a speed-up of 2-4 is expected on the same machine in this relatively small problem. Using 6 processes on GASFLOW-server, it took 39.6 hours to finish the calculation. With a single Intel I7 CPU with frequency of 3.07 GHz, it took GASFLOW-SEQ 90 hours to calculate the problem. It should be noted that for such a small problem there are only roughly 8,000 cells on each process in the GASFLOW-MPI calculation, and the communication effort starts to become significant compared with the computational effort. The pressure and temperature were compared in Figure 16 (red: GASFLOW-SEQ, green: GASFLOW-MPI). Due to the differences of the recombiner models in the two GASFLOW versions, there are slight discrepancies during 11,000 s and 18,000 s. In general, GASFLOWMPI is able to reproduce the previous calculation executed by GASFLOW-SEQ in the APR1400 analysis with complex geometry and physics. 5.4 Speed-up of EPR simulation EPRTM is a third generation PWR designed by Areva GmbH [32]. The GASFLOW containment model was developed in a cylindrical mesh with 30 radial, 120 azimuthal, and 83 axial nodes. The containment model has a free gas volume of 78,751 m3, as shown in Figure 17Figure 18 The Page 10 of 14
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
total number of cells is 298,800. A postulated SBLOCA with break at the hot leg was simulated until 8,000 second. The vector supercomputer NEC SX-8R at KIT was used by Areva GmbH to calculate the problem, and it took around 634 hours. With 32 processes on our internal server, it took GASFLOW-MPI 199 hours to finish the simulation. It should be noted SX-8R is roughly 4 times faster than a typical PC with Intel I7 CPU which was used by many GASFLOW-SEQ users. The speed-up of GASFLOW-MPI is highly dependent on the configuration of the high performance cluster and the size of the computed problem. The speed-up can be further increased by using more powerful high performance cluster and optimizing the GASFLOW-MPI code. 6
CONCLUSIONS AND FUTURE WORK
GASFLOW solves 3-D transient, two-phase, compressible Navier-Stokes equations for multispecies using a proven semi-implicit pressure-based algorithm of ICE’d-ALE for all-speed flows [15]. GASFLOW has been fully validated with international experimental benchmarks, and widely applied in the hydrogen safety analysis involving NPP containments. The simulation results of the GASFLOW code have been widely accepted by the nuclear authorities in several European and Asian countries [12, 17-19, 31-41]. GASFLOW-MPI, which utilizes the data structure and linear solvers in PETSc, is the parallel version of GASFLOW. GASFLOW-MPI is a successful application case of PETSc in parallelization of a sequential CFD code which solves 3-D two-phase compressible N-S equations using staggered grid. Very good agreement as well as significant speedup has already been demonstrated for the results from various benchmark problems with GASFLOW-MPI and GASFLOW-SEQ codes. Parallelization of GASFLOW based on the PETSc library adds tremendous value to the numerical simulation of hydrogen safety analysis in large scale industrial facilities. It enables the creation of large scale and high-fidelity models which provide more accurate and detailed predictions. GASFLOW-MPI will help to understand the physical behaviors in NPP containments under severe accidents with considerably greater level of details. It provides to the industrial users higher level of confidence in the design of the risk mitigation systems and helps to fulfill the requirements of the licensing authorities. Moreover, it can efficiently reduce the cost of computational resources required in an engineering project. GASFLOW-MPI has been and will be further tested using a Standard Test Matrix of problems in five categories: (1) Feature or verification tests for the computer science aspects of the code; (2) Functional tests for code algorithms, equations, logic paths, and decision points; (3) Comparisons of computational results with analytical solutions; (4) Comparisons of computational results with experimental data, as shown in Table I; (5) Comparisons of computational results with previous results in large scale industrial applications, as shown in Table II. Advanced features, such as load balancing, immersed boundary and adaptive mesh refinement, require further considerations. GASFLOW-MPI will continue its further development targeting a high performance, fully validated CFD code for the thermal hydraulics and safety analyses in the NPP containments and other large scale industrial applications.
Page 11 of 14
Jianjun Xiao et al.
Figure 13. 3D Geometry model of Konvoi Plant for GASFLOW with source and recombiner positions (Courtesy of P. Royl)
Figure 15. GASFLOW containment model for APR1400 with Passive Autocatalytic Recombiners (PARs)
Figure 14. Comparisons of MBLOCA with/without radiation heat transfer using GASFLOW-SEQ on KIT SX-8R and GASFLOW-MPI on GASFLOW-server
Figure 16. Comparisons of LBLOCA using GASFLOW-SEQ and GASFLOW-MPI
Figure 17. Schematic of the EPRTM Figure 18. 3-D EPRTM geometry model for containment. (Courtesy of H. Dimmelmeier) GASFLOW simulation.
Page 12 of 14
GASFLOW-MPI: a parallel CFD code for safety analysis in NPP containment
7
ACKNOWLEDGMENTS
The authors would like to thank Barry Smith, Matthew Knepley, Jed Brown and Satish Balay of the PETSc development team for useful discussions and support. The authors also acknowledge Harald Dimmelmeier from Areva GmbH for providing input for GASFLOW-MPI testing. 8 1. 2. 3. 4. 5. 6.
7.
8.
9.
10.
11.
12. 13. 14. 15. 16. 17.
18.
19. 20.
REFERENCES
Breitung, W., et al., SOAR on containment thermal hydraulics and hydrogen distribution. NEA/CSNI/2000/7. OECD, Paris, France (1999). Breitung, W., Royl, P., Procedure and tools for deterministic analysis and control of hydrogen behavior in severe accidents, Nuclear Engineering and Design, Volume 202, Issues 2–3, 249–268, (2000). IAEA, Fukushima Daiichi Status Report, Vienna, (2011). Travis, J. R., "HMS: A Model for Hydrogen Migration Studies in LWR Containments," Second Intern. Workshop on the Impact of Hydrogen on Water Reactor Safety, Albuquerque, New Mexico, October 3-7, 1982. Travis, J. R., "HMS: A Computer Program for Transient, Three-Dimensional Mixing Gases," Los Alamos National Laboratory report LA-10267-MS (NUREG/CR-4020) (November 1984). Travis, J. R., K. L. Lam, and T. L. Wilson, GASFLOW: A Three-Dimensional Finite-Volume Fluid-Dynamics Code for Calculating the Transport, Mixing, and Combustion of Flammable Gases in Geometrically Complex Domains, LA-UR-94-2270, July 11, 1994 Nichols, B. D., C. Mueller, G. A. Necker, J. R. Travis, J. W. Spore, K. L. Lam, P. Royl, R. Redlinger and T. L. Wilson, GASFLOW: a computational fluid dynamics code for gases, aerosols and combustion, Volume 1: Theory and Computational Model, LA-13357-M, FZKA-5994, Vol. I, Manual, (1998). Nichols, B. D., C. Mueller, G. A. Necker, J. R. Travis, J. W. Spore, K. L. Lam, P. Royl, R. Redlinger and T. L. Wilson, GASFLOW: a computational fluid dynamics code for gases, aerosols and combustion, Volume 2: User’s Manual, LA-13357-M, FZKA-5994, Vol. II, Manual, (1998). Müller, C., E. D. Hughes, G. F. Niederauer, H. Wilkening, J. R. Travis, J. W. Spore, P. Royl and W. Baumann, GASFLOW: a computational fluid dynamics code for gases, aerosols and combustion, Volume 3: Assessment Manual, LA-13357-M, FZKA-5994, Vol. III, Manual, (1998) J. Xiao, J. R. Travis, T. Jordan, P. Royl, G. A. Necker, R. Redlinger, and A. Svishchev, GASFLOW-III: A Computational Fluid Dynamics Code for Gases, Aerosols and Combustion, Volume 1: Theory and Computational Model, Karlsruhe Institute of Technology report, Revision 3.5, (2014). J. Xiao, J. R. Travis, T. Jordan, P. Royl, G. A. Necker, R. Redlinger, and A. Svishchev, GASFLOW-III: A Computational Fluid Dynamics Code for Gases, Aerosols and Combustion, Volume 2: User’s Manual, Karlsruhe Institute of Technology report, Revision 3.5, (2014). P. Royl, R., J. R. Travis and Jongtae Kim, GASFLOW-III: A Computational Fluid Dynamics Code for Gases, Aerosols and Combustion, Volume 3: Assessment Manual, Karlsruhe Institute of Technology report, (2014). Satish Balay, et al., PETSc Users Manual, ANL-95/11 - Revision 3.5, Argonne National Laboratory, (2014). Breitung, W., et al., State of the Art Report on Flame Acceleration and Deflagration-to-Detonation Transition in Nuclear Safety NEA/CSNI/R (2000)7. OECD Nuclear Energy Agency, Issy-les-Moulineaux, France, (2000). Hirt, C.W., Amsden, A.A., and Cook, J.L., An Arbitrary Lagrangian-Eulerian Computing Method for All Flow Speeds, J. Comp. Phys. 14, 227, (1974). Harlow, F.H. and Amsden, A.A., A numerical fluid dynamics calculation method for all flow speeds, J. Comp. Phys., 8, p. 197, (1971). P. Royl, C. Müller, J. R. Travis, T. Wilson, ”Validation of GASFLOW for Analysis of Steam/Hydrogen Transport and Combustion Processes in Nuclear Reactor Containments,” Procs 13th Conference on Structural Mechanics in Reactor Technology, Porto Alegre, RS, Brazil, (1995). P. Royl, J. R. Travis, E. A. Haytcher, and H. Wilkening, "Analysis of Mitigating Measures during Steam/Hydrogen distributions in Nuclear Reactor Containments with the 3D Field Code GASFLOW," presented at the OECD/NEA CSNI Workshop on the Implementation of Hydrogen Mitigation Techniques, Winnipeg, Canada, (1996). P. Royl, J. R. Travis, W. Breitung, “Benchmarking of the 3D CFD Code GASFLOW with Containment Thermal Hydraulic Tests from HDR and ThAI”, procs IAEA-OECD CFD4NRS Conference, (2006). P. Royl, G. Necker, J. W. Spore, J. R. Travis, ”3D Analysis of Hydrogen Recombination Experiments in the Battelle Model Containment with the GASFLOW Code.” Procs Jahrestagung Kerntechnik, Munich, (1998). Page 13 of 14
Jianjun Xiao et al.
21. P. Royl, J. R. Travis, W. Breitung, “Analyses of Containment Experiments with GASFLOW”, procs Nureth-10 Conference, Seoul, Korea, (2003). 22. H.J. Allelein, et al.: “International Standard Problem ISP-47 on Containment Thermal Hydraulics”, OECD final report, (2006). 23. P. Royl, et. al., “GASFLOW Analysis of Steam/Hydrogen Mixing with Nitrogen in the ORCD-NEA THAI HM2 Benchmark”, NURETH-13, Kanazawa City, Japan, (2009). 24. J. Xiao, J. R. Travis, “How critical is turbulence modeling in gas distribution simulations of large-scale complex nuclear reactor containment?”, Annals of Nuclear Energy, Volume 56, Pages 227–242, (2013). 25. J. Malet, et. al., “Achievements of spray activities in nuclear reactor containments during the last decade”, Annals of Nuclear Energy, 74:134-142, (2014). 26. P. Royl, et.al., “GASFLOW Validation with Panda Tedts from the OECD SETH Benchmark Covering Steam/Air and Steam/Helium/Air Mixtures”, Science and Tecnology of Nuclear Installations, (2009). 27. J. Malet, et. al., “Gas entrainment by one single French PWR spray, SARNET-2 spray benchmark”, Nuclear Engineering and Design, Volume 282, Pages 44–53, (2015). 28. P. Royl, J. R. Travis, “Simulation of Hydrogen Transport with Mitigation Using the 3D Field Code GASFLOW”, Procs. 2nd International Conference on Advanced Reactor Safety, Orlando, Florida, (1997). 29. P. Royl, J. R. Travis, W. Breitung, “ Modelling and Validation of Catalytic Hydrogen Recombination in the 3D CFD Code GASFLOW II”, procs IAEA-OECD CFD4NRS Conference Munich, (2006). 30. Teng Tu, Po Hu, “Test Analysis of the Combustion Mechanism of Hydrogen-Air Mixture”, Atomic energy science and technology, (2014, in Chinese). 31. S. Benz, et al., “Analysis of a hypothetical Loss of Coolant Accident in a Konvoi type NPP by GASFLOW and COCOSYS”, Procs Jahrestagung Kerntechnik, Berlin, (2013). 32. Dimmelmeier, H., Eyink, J., Movahed, M.A., Computational validation of the EPR combustible gas control system. Nuclear Engineering and Design 249, 118–124, (2012). 33. P. Royl, et al., “Simulation of hydrogen transport with mitigation using the 3D field code gasflow”, Proceedings of the international topical meeting on advanced reactors safety: Volume 1, 578-588, (1997). 34. M. Heitsch, et.al., “CFD evaluation of hydrogen risk mitigation measures in a VVER-440/213 containment”, Nuclear Engr. Design, Vol. 240, pp. 385-396, (2010). 35. J. Kim, S. Hong, “Analysis of hydrogen flame acceleration in APR1400 containment by coupling hydrogen distribution and combustion analysis codes”, Progress in Nuclear Energy, Volume 78, Pages 101–109, (2015). 36. J. Kim, et al., “Hydrogen Safety Analysis of the OPR1000 Nuclear Power Plant during a Severe Accident by a Small-Break Loss of Coolant”, Proceedings of the KNS spring meeting, (2009). 37. Unjang Lee, et al., 'Three Dimensional Analysis of the Steam-Hydrogen Distribution from a Hypothetical Small Break Severe Loss of Coolant Accident in a VVER 1000 Type reactor Containment Using GASFLOW II ,' NURETH-10, Seoul, Korea, (2003). 38. J. Xiong, et al., “CFD Application to Hydrogen Risk Analysis and PAR Qualification”, Science and Technology of Nuclear Installations, Volume 2009 (2009) 39. J. Xiao, et al., “Numerical analysis of hydrogen risk mitigation measures for support of ITER licensing”, Fusion Engineering and Design, 85(2):205-214, (2010). 40. R. Zhang, et al., “The CFD Analysis of Hydrogen Behavior in HTR-PM during the Water ingress Accident Caused by Double-ended Guillotine Break of Two Steam Generator Heating Tubes”, Proceedings of the 2014 22nd International Conference on Nuclear Engineering, Prague, Czech Republic, (2014). 41. T. Jordan, “Use of The KIT HYCODES System with a Conservative Source for the Simulation of Hydrogen Distribution and Combustion with/without Spray in a Large Break LOCA in the Korean APR1400”, Jahrestagung Kerntechnik Berlin, (2015). 42. Sod G.A., “A survey of Several Finite Difference Methods for Systems of Nonlinear Hyperbolic Conservation”, Journal of Computational Physics, vol. 27, pp. 1-31, (1978). 43. U. Ghia, High-Re solutions for incompressible flows using the Navier-Stokes equations and a multigrid method, Journal of Computational Physics , 48, 387-411, (1982).
Page 14 of 14
