Parallel Multiphysics Algorithms and Software for Computational Nuclear Engineering. D. Gaston, G. Hansen, S. Kadioglu, D.A. Knoll,. C. Newman, H. Park, ...
Parallel Multiphysics Algorithms and Software for Computational Nuclear Engineering D. Gaston, G. Hansen, S. Kadioglu, D.A. Knoll, C. Newman, H. Park, C. Permann*, W. Taitano Multiphysics Methods Group and *Center for Advanced Modeling and Simulation Idaho National Laboratory Idaho Falls, Idaho
Outline • Introduction/Motivation • Numerical Method and Framework – Jacobian-free Newton-Krylov (JFNK) Methods – Physics-based Preconditioning – MOOSE
• Applications – Gas-cooled Reactor Simulation, PRONGHORN – Fuels simulation, BISON
• JFNK as a multiscale solver • Conclusions • Not discussing multiple mesh issues, although equally important
Multiphysics Computational Nuclear Engineering • Reactor Simulation – Coupling of coolant flow, heat transfer, neutronics, structural mechanics … – Applications include steady-state design, design transients, and accident analysis
• Fuels Simulation – Coupling of nonlinear thermomechnics, contact, chemistry, fission product behavior, … – Again, applications include steady-state design, design transients, and accident analysis
• Tight vs. Loose coupling – Often, single physics legacy codes are coupled to do multiphysics calculations. – This can produce time integration errors and stability issues – We are developing tightly coupled application codes with modern algorithms
Motivation for Jacobian-Free Newton-Krylov Methods • Tightly coupled methods without the Jacobian • Re-use of operator splitting algorithms (legacy solver) as preconditioner (physics-based). We are using loose coupling as preconditioner ! • Algorithms can fit nicely into an object-oriented multiphysics framework allowing for rapid development • Improved access to adjoint-based methods for adaptation and UQ, data assimilation, and PDEconstrained optimization (modern design)
JFNK Basics •Using Newton’s method to solve the nonlinear system
•The resulting linear system and nonlinear iteration are
•Using a Krylov method (GMRES) to solve the linear system only requires a matrix-vector product, not the matrix by its self
•This matrix-vector product (for generic v) can be approximated by,
Preconditioning JFNK •Preconditioning is the KEY for efficient application to multiphysics engineering problems.
•Right-preconditioned matrix-free version is:
• Traditional Preconditioning •Step 1: Choosing the matrix, •Step 2: Approximating , typically ILU or Schwarz-ILU based methods, or perhaps systems multigrid • Using Multilevel methods with standard smoothers here can be challenging for multiphysics (i.e. stiff ) systems
Physics-Based Preconditioning (1 of 2) •Step 1: Choosing the matrix,
!
Instead M is some approximate combination of linear operators (scalar elliptic problems) •Step 2: Approximating M -1 is a two step process 1.Ordering represent some standard operator split approximation to a time step with the approximate inversion of each sub-system 2.Use modern parallel multilevel solvers to approximately invert each elliptic operator (sub-system) • Now using multilevel methods with standard smoothers can be very effective since we have scalar elliptic problems
Physics-Based Preconditioning (2 of 2) •Simple two Equation system
•Block Jacobi (most simple picture), block inversions with multilevel methods
• Block GS
•Many other related options / extensions have been used. All represent some form of a legacy solver / time step, Semi-implicit methods, SIMPLE, …
JFNK: Summary and Algorithm
Solving nonlinear system implicitly without forming expensive Jacobian matrix. Matrix-vector product required by the Krylov Physics-based method is approximated by the preconditioning finite difference.
MOOSE: Multiphysics Object Oriented Simulation Environment • Requirements: – – – – –
3D (and 2D (and 1D)) Massively Parallel and Portable Fully Coupled and Implicit (JFNK and Physics-based Preconditioning) Advanced solution strategies (Adaptivity, etc.) Flexible Physics Interface and Materials Database
• Currently supports Finite Elements and Adaptivity • Adjoint methods under development • Has allowed for rapid multiphysics engineering application code development. Started 13 months ago
MOOSE Platform •
Plug-and-play modules – Simplified coupling
•
MOOSE Physics Interface conceals framework complexity
•
Framework provides core set of common services – libMesh: http://libmesh.sf.net
•
Solver Interface abstracts specific solver implementations. – Common interface to linear and nonlinear solvers – More flexible
•
Utilize state-of-the-art linear and nonlinear solvers – Robust solvers are key for “ease of use”
&
% • •(
• &
$
! #
PRONGHORN Project •Supports Very High Temperature Reactor (VHTR) analysis in DoE’s NGNP program •Helium cooled, graphite moderated • Both pebble-bed and prismatic core concepts being considered •Previous multiphysics analysis has been done with loosely coupled legacy codes •PRONGHORN project is 9 months old
PRONGHORN: Initial Equation Set (1 of 2) • Pebble-bed concept: Thermal-fluid model is currently Darcy-like flow model. Initial coupled model is required to do the PBMR 400 suite of published benchmark problems.
• Momentum ( W is porous media friction model) • Continuity • Fluid Energy
PRONGHORN: Initial Equation Set (2 of 2) • Solid Energy • Neutronics Model (Multigroup diffusion)
• Material properties are generally functions of fluid and solid state variables. • Extensions to more sophisticated thermal-flow model and neutronics model in progress
Physics-Based Preconditioner: Example Original System
Linearize material property Preconditioning System Down scatter Fission Heat Heat Transfer Buoyancy Force
Algorithm Result: Newton vs. Picard iterations
nonlinear iterations
CPU time
Newton
7
27sec
Picard
14
54sec
Accuracy Result: higher- and lower-order method Numerical diffusion due to 1st order time integration (~15 ms)
Error due to explicit coupling (>20%) t = 1.5 ms tfine = 0.5 ms
Reference
3134.47C (Max T@ 3s)
BE1 explicit
3195.73
BDF2 explicit
3273.69
BE1 implicit
3112.89
BDF2 implicit
3137.48
Fully coupled criticality search of PBMR 400 Block Jacobi
Block Gauss-Seidel
CPU time
700
432
Average (total) GMRES
37.6 (1390)
22.4(693)
keff and max temperature as function of total power.
Nuclear Fuel Issues
BISON Project •
Project can support fuel performance modeling for most reactor concepts, including Light Water Reactors (LWRs).
•
Project is 13 months old.
•
Some specific fuels issues:
•
Details of fuel pellet / cladding interaction
•
Multiscale thermal transport
•
Fission gas formation / migration
•
Restructuring, constituent migration
•
Damage – radiation, cracking, corrosion
BISON: Initial Equation Set (1 0f 2) • Linear Mechanics
With
and
BISON: Initial Equation Set (2 0f 2) • Heat Conduction • Oxygen Diffusion
• Models for: Contact, Fission gas behavior, mechanical properties, thermal conductivity
• Extensions to nonlinear mechanics, implicit contact, and crack models in progress
BISON: Multiple time scale transient
BISON: Pellet-Clad Interaction
JFNK Scale Bridging: VERY NEW • Engineering scale problem defines nonlinear function to be solved by JFNK •Lower length scale system is solved (or relaxed) self-consistently within the engineering scale nonlinear function •Solution of engineering scale is used to accelerate lower length scale solver. The lower-length scale solver only relaxes high frequency solution structure •Solution of the lower length problem is used to provide closure (or increased accuracy) to the engineering scale •Current examples include acceleration of Boltzmann solver for reactor neutronics and inclusion of lower-length scale effects in fuel thermal transport (M. Tonks Poster, Tuesday)
Where are we going - JFNK LLS-informed Continuum-level Simulation F(u) is the vector-valued function of nonlinear residuals
Given the phonon distribution function, one can evaluate the local heat flux (thermal conductivity k is not needed). This heat flux can then be used to complete the energy equation residual.
The energy equation is solved at the engineering scale for temperature using JFNK
Solve the phonon transport equation for the phonon distribution function at the mesoscale. This equation is driven by the local temperature and accounts for grain boundary and radiation damage effects. This transport equation is solved inside the energy equation residual function.
BISON: Parallel Scaling on Multiscale Problem
Conclusion • Multiphysics simulation is the next frontier in computational nuclear engineering.
• Jacobian-free Newton-Krylov methods and physics-based preconditioner allow tightly coupled multiphysics simulations.
• Multiphysics Object-Oriented Software Environment, MOOSE, is allowing for rapid development of , 3-D, parallel, multiphysics engineering application tools.
• NOTE: Also using JFNK to significantly accelerate eigenvalue iteration required for steady-state design.
