Building MFIX-Exa

This page gives the generic instructions for building MFIX-Exa using gmake and cmake. The instructions assume you have already downloaded and unpacked the MFIX-Exa code into a local mfix directory. If not, please visit the download page first. For HPC cluster specific instructions, please refer to Building on HPC Systems.

Building with gmake

If you want to use gmake to build MFIX_Exa, you will need to clone amrex and AMReX-Hydro into a local directories:

> git clone https://github.com/AMReX-Codes/amrex.git
> git clone https://github.com/AMReX-Codes/AMReX-Hydro.git
> cd mfix/exec

Then, edit the GNUmakefile (or set an environment variable) to define AMREX_HOME and AMREX_HYDRO_HOME to be the path to the directories where you have put amrex and AMReX-Hydro. Note that the default locations of AMREX_HOME and AMREX_HYDRO_HOME are in the subprojects directory, which is where a cmake configuration may clone these repositories. Other options that you can set include

Option name

Description

Possible values

Default value

COMP

Compiler (gnu or intel)

gnu / intel

gnu

USE_MPI

Enable MPI

TRUE / FALSE

FALSE

USE_OMP

Enable OpenMP

TRUE / FALSE

FALSE

USE_CSG

Enable CSG geometry file support

TRUE / FALSE

FALSE

USE_CUDA

Enable CUDA GPU support

TRUE / FALSE

FALSE

USE_HIP

Enable HIP GPU support

TRUE / FALSE

FALSE

USE_DPCPP

Enable DPCPP GPU support

TRUE / FALSE

FALSE

USE_HYPRE

Enable HYPRE support

TRUE / FALSE

FALSE

DEBUG

Create a DEBUG executable

TRUE / FALSE

FALSE

PROFILE

Include profiling info

TRUE / FALSE

FALSE

TRACE_PROFILE

Include trace profiling info

TRUE / FALSE

FALSE

COMM_PROFILE

Include comm profiling info

TRUE / FALSE

FALSE

TINY_PROFILE

Include tiny profiling info

TRUE / FALSE

FALSE

DIM

Dimensionality of problem

2 / 3

3

Note

Do not set both USE_OMP and USE_CUDA/HIP/DPCPP to true.

Then type

> make -j

An executable will appear; the executable name will reflect some of the build options above.

Building with cmake

CMake build is a two-step process. First cmake is invoked to create configuration files and makefiles in a chosen directory (builddir). Next, the actual build is performed by invoking make from within builddir. If you are new to CMake, this short tutorial from the HEP Software foundation is the perfect place to get started with it.

The CMake build process for MFIX-Exa is summarized as follows:

>> mkdir /path/to/builddir
>> cd    /path/to/builddir
>> cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix
>> make

In the above snippet, [options] indicates one or more options for the customization of the build, as described later in this section. If the option CMAKE_BUILD_TYPE is omitted, CMAKE_BUILD_TYPE=Release is assumed.

There are two modes to build MFIX-Exa with cmake:

o SUPERBUILD (recommended): The AMReX and AMReX-Hydro git repos are cloned and built as part of the MFIX-Exa build process and placed in the mfix/subprojects directory. Each of these repos can be manipulated like a regular git repo (you can change branches, hashes, add remotes, etc.) This method is strongly encouraged as it ensures that the configuration options for MFIX-Exa are compatible with the AMReX and AMReX-Hydro hashes that are checked out.

o STANDALONE: MFIX-Exa source code is built separately and linked to existing AMReX and AMReX-Hydro repos. This is ideal for continuous integration severs (CI) and regression testing applications. AMReX and AMReX_Hydro library versions and configuration options must meet MFIX-Exa requirements.

Note

MFIX-Exa requires CMake 3.20 or higher.

STANDALONE instructions

Building AMReX

Clone AMReX from the official Git repository. Note that the only branch available is development:

> git clone https://github.com/AMReX-Codes/amrex.git

Next, configure, build and install AMReX as follows:

> cd amrex
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DAMReX_PARTICLES=yes -DAMReX_EB=yes -DAMReX_PLOTFILE_TOOLS=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute_path_to_amrex_installdir ..
> make install

The options AMReX_PARTICLES=yes, AMReX_EB=yes and AMReX_PLOTFILE_TOOLS=yes are required by MFIX-Exa. [other amrex options] in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the AMReX user guide for more details on building AMReX with CMake.

Building MFIX-Exa

Clone and build MFIX-Exa:

> cd mfix/
> mkdir build
> cd build
> cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
> make -j

Passing -DAMReX_ROOT=/absolute/path/to/amrex/installdir instructs CMake to search /absolute/path/to/amrex/installdir before searching system paths for an available AMReX installation. AMReX_ROOT can also be set as an environmental variable instead of passing it as a command line option. [mfix options] indicates any of the configuration option listed in the table below.

Table 1 MFIX-Exa configuration options

Option name

Description

Possible values

Default value

CMAKE_CXX_FLAGS

User-defined C++ flags

valid C++ compiler flags

None

CMAKE_CUDA_FLAGS

User-defined CUDA flags

valid CUDA compiler flags

None

MFIX_MPI

Enable build with MPI

no/yes

yes

MFIX_OMP

Enable build with OpenMP

no/yes

no

MFIX_GPU_ BACKEND

On-node, accelerated GPU backend

NONE,SYSCL,CUDA,HIP

NONE

MFIX_HYPRE

Enable HYPRE support

no/yes

no

MFIX_FPE

Build with Floating-Point Exceptions checks

no/yes

no

MFIX_CSG

Build with CSG support

no/yes

no

MFIX_MPI_ THREAD_MULTIPLE

Concurrent MPI calls from multiple threads

no/yes

no

Few more notes on building MFIX-Exa with cmake

The system defaults compilers can be overwritten as follows:

> cmake -DCMAKE_CXX_COMPILER=<c++-compiler> [options]  ..

When building on a platform that uses the module utility, use either the above command (with full path to the compilers) or the following:

> cmake -DCMAKE_CXX_COMPILER=CC [options] ..

MFIX-Exa uses the same compiler flags used to build AMReX, unless CMAKE_CXX_FLAGS is explicitly provided, or the environmental variable CXXFLAGS is set.

For GPU builds, MFIX-Exa relies on the AMReX GPU build infrastructure . The target architecture to build for can be specified via the AMReX configuration option -DAMReX_CUDA_ARCH=<target-architecture>, or by defining the environmental variable AMREX_CUDA_ARCH (all caps). If no GPU architecture is specified, CMake will try to determine which GPU is supported by the system.