MFIX-Exa

A high-performance, massively parallel, machine agnostic multiphase flow code for the exascale computing era. MFIX-Exa is built using the AMReX framework which provides iterators, linear solvers, parallel communication routines, and other utilities that support highly efficient operations on structured grid and particle data. MFIX-Exa performance portability is inherited from AMReX’s flexible MPI+X hybrid parallelism strategy allowing MFIX-Exa to offload to Nvidia, AMD, and Intel GPU accelerators. The code has been deployed out to at least 512 GPUs on the DOE Leadership Computing Facility (LCF) machines, including: Frontier (OLCF), Summit (OLCF), Perlmutter (NERSC), and Polaris (ALCF).
MFIX-Exa

Background Development

In 2016, the US DOE formed the Exascale Computing Project (ECP) to usher the nation into the exascale computing era by bringing together HPC research, development and deployment activities. ECP resulted in the world’s first exascale benchmark carried out on OLCF’s Frontier, clocking in at approximately 1.1 exaFLOPS. ALCF is set to deliver Aurora in 2023 which is expected to nearly double the compute capability of Frontier. In addition to just procuring the hardware, the DOE long recognized that computational codes and workflows also needed support to fully utilize the complex, heterogeneous architectures that would likely find their way into the first exascale machines. Fortunately, a team led by NETL and LBNL was selected as an ECP Energy Application code to develop an exascale-capable CFD-DEM code for the simulation of gas-solid particle flows. The result of this ECP funded collaboration was the MFIX-Exa code.

ECP

Capabilities

MFIX-Exa is a CFD-DEM and PIC code featuring:

  • The fluid solver is a modern implementation of a low-Mach number projection method with a Godunov time advancement scheme and supports Incompressible and Ideal-gas (weakly compressible) continuity constraints
  • The DEM model uses a linear-spring dashpot soft-sphere type collision model
  • The PIC model uses an iterative solids stress model to minimize over packing
  • Enthalpy transport, species equations and chemical reactions are (optionally) solved for thermal problems without or with heterogeneous chemical reactions
  • Complex geometries are resolved by an embedded boundary (EB), on which fluid and particle flow boundary conditions are directly applied
  • Embedded boundaries are generated from a user provided constructive solids geometry (CSG) file, which can be created using an external CAD program such as the open-source OpenSCAD program
  • MFIX-Exa offers a novel pic2dem application which can be used to bootstrap initial conditions for a CFD-DEM simulation from a lower fidelity PIC simulation
  • MFIX-Exa has a wide variety of data reduction methods in the form of user monitors and supports in situ visualization with Ascent

MFIX-Exa's Challenge Problem

MFIX-Exa’s ECP challenge problem is a demonstration simulation of NETL’s 50kW pilot-scale chemical looping reactor with the CFD-DEM model. The animation shows a start-up PIC simulation on a coarse 4 mm fluid mesh with a statistical weight of 512. The pic2dem app was applied to the end state of this simulation to create the challenge problem initial conditions on a high-resolution 400 micron fluid mesh containing over 5 billion particles. A short DEM simulation was continued on OLCF’s Frontier utilizing over 62,000 AMD MI250X GCDs.

Limitations

All ECP application teams identified a challenge problem at the beginning of the project to assess code performance and utilization of exascale resources. MFIX-Exa’s challenge problem, shown above, has driven many of the development choices to date. Hence, if your target application is like the CLR above, MFIX-Exa may be well-suited to your problem. However, MFIX-Exa does not (and may not ever) contain all of the wide-reaching physics, sub-models and user features as the classic MFiX code. Some notable limitations potential users should consider are listed below.

  • Primary code development activities are complete; however, the codebase is not as mature as MFiX classic. Releases are simply monthly tags of the continually evolving develop branch.
  • Currently, there is no graphical user interface (GUI) but plans are being formulated for future GUI development.
  • Only spherical particles are considered.
  • The fluid solver is low-Mach, weakly compressible and CFL limited. Rarified or high-speed flows are not expected to preform well physically nor computationally.
  • Heterogenous reactions employ a type of shrinking core (constant radius) model, i.e., changes in mass result in changes in density, not changes in volume.
  • Homogeneous chemical reactions are currently neglected and there is no stiff chemistry solver (direct integrator).
  • There is no radiation model.

In addition to the aforementioned limitations, potential users are cautioned that MFIX-Exa and AMReX are high-performance codes, therefore development work is more complex than what you may be used to with MFiX classic. Additionally, while MFIX-Exa has a continuous integration harness, a regression test suite, and has been validated on several classic CFD-DEM benchmark experiments, it has not undergone the same rigorous verification and validation (V&V) process of MFiX classic. Further, the broad reach of the region of validity for the classic MFiX codebase was largely provided by the wide-ranging user community which is only just beginning for MFIX-Exa.

If you are undeterred by these warnings and believe MFIX-Exa is well-suited for you and your particle flow application, we encourage you to follow the steps below and join our growing userbase.

Getting Started

Disclaimer
Privacy Policy
Web Policies
FOIA/Privacy Act
Vulnerability Disclosure Program
Contact NETL