# Building MFIX-Exa

This page gives the generic instructions for building MFIX-Exa using gmake and cmake. For HPC cluser specific instructions, please refer to Building on HPC Sytems.

## Building with gmake

If you want to use gmake to build MFIX_Exa, you will need to clone amrex and AMReX-Hydro into a local directories, and also clone mfix:

    > git clone https://github.com/AMReX-Codes/amrex.git
    > git clone https://github.com/AMReX-Codes/AMReX-Hydro.git
    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    > cd mfix/exec

Then, edit the GNUmakefile (or set an environment variable) to define AMREX_HOME and AMREX_HYDRO_HOME to be the path to the directories where you have put amrex and AMReX-Hydro. Note that the default locations of AMREX_HOME and AMREX_HYDRO_HOME are in the subprojects directory, which is where a cmake configuration may clone these repositories. Other options that you can set include

Option name | Description | Possible values | Default value
--- | --- | --- | ---
COMP | Compiler (gnu or intel) | gnu / intel | gnu
USE_MPI | Enable MPI | TRUE / FALSE | FALSE
USE_OMP | Enable OpenMP | TRUE / FALSE | FALSE
USE_CSG | Enable CSG geometry file support | TRUE / FALSE | FALSE
USE_CUDA | Enable CUDA GPU support | TRUE / FALSE | FALSE
USE_HIP | Enable HIP GPU support | TRUE / FALSE | FALSE
USE_DPCPP | Enable DPCPP GPU support | TRUE / FALSE | FALSE
USE_HYPRE | Enable HYPRE support | TRUE / FALSE | FALSE
DEBUG | Create a DEBUG executable | TRUE / FALSE | FALSE
PROFILE | Include profiling info | TRUE / FALSE | FALSE
TRACE_PROFILE | Include trace profiling info | TRUE / FALSE | FALSE
COMM_PROFILE | Include comm profiling info | TRUE / FALSE | FALSE
TINY_PROFILE | Include tiny profiling info | TRUE / FALSE | FALSE
DIM | Dimensionality of problem | 2 / 3 | 3

Note

Do not set both USE_OMP and USE_CUDA/HIP/DPCPP to true.

Then type

    > make -j

An executable will appear; the executable name will reflect some of the build options above.

## Building with cmake

CMake build is a two-step process. First `cmake` is invoked to create configuration files and makefiles in a chosen directory (`builddir`). Next, the actual build is performed by invoking `make` from within `builddir`. If you are new to CMake, this short tutorial from the HEP Software foundation is the perfect place to get started with it.

The CMake build process for MFIX-Exa is summarized as follows:

    >> mkdir /path/to/builddir
    >> cd    /path/to/builddir
    >> cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix
    >> make

In the above snippet, `[options]` indicates one or more options for the customization of the build, as described later in this section. If the option `CMAKE_BUILD_TYPE` is omitted, `CMAKE_BUILD_TYPE=Release` is assumed.

There are two modes to build MFIX-Exa with cmake:

o SUPERBUILD (recommended): The AMReX and AMReX-Hydro git repos are cloned and built as part of the MFIX-Exa build process and placed in the `mfix/subprojects` directory. Each of these repos can be manipulated like a regular git repo (you can change branches, hashes, add remotes, etc.) This method is strongly encouraged as it ensures that the configuration options for MFIX-Exa are compatible with the AMReX and AMReX-Hydro hashes that are checked out.

o STANDALONE: MFIX-Exa source code is built separately and linked to existing AMReX and AMReX-Hydro repos. This is ideal for continuous integration severs (CI) and regression testing applications. AMReX and AMReX_Hydro library versions and configuration options must meet MFIX-Exa requirements.

Note

MFIX-Exa requires CMake 3.20 or higher.

### SUPERBUILD Instructions (recommended)

By default MFIX-Exa CMake looks for an existing AMReX installation on the system. If none is found, it falls back to SUPERBUILD mode. In this mode, MFIX-Exa CMake inherents AMReX CMake targets and configuration options, that is, MFIX-Exa and AMReX are configured and built as a single entity

Assuming no valid AMReX installation is present on the target system, and `AMReX_ROOT` is not set (see STANDALONE instructions), the following code will build MFIX-Exa in SUPERBUILD mode:

    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    > cd mfix
    > mkdir build
    > cd build
    > cmake [mfix options] [amrex options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..
    > make -j

`[amrex options]` in the snippet above is a list of any of the AMReX configuration options listed in the AMReX user guide, while `[mfix options]` is any of the configuration options listed here.

For example, to enable AMReX profiling capabilities in MFIX_Exa, configure as follows:

    > cmake [mfix options] -DAMReX_TINY_PROFILE=yes -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..

#### Working with the AMReX submodule

SUPERBUILD mode relies on a git submodule to checkout the AMReX git repository. If the AMReX submodule is not initialized, SUPERBUILD mode will initialize it and checkout the AMReX commit the submodule is pointing to. Instead, if the AMReX submodule has already been manually initialized and a custom commit has been checked out, SUPERBUILD mode will use that commit. For MFIX-Exa development or testing, you may need to build with a different branch or release of AMReX.

The `subprojects/amrex` directory is a Git repo, so use all standard Git commands. Either `cd subprojects/amrex` to run Git commands in the `amrex` directory, or use `git -C subprojects/amrex` in the MFIX-Exa repo. For instance, to build with the `my-amrex-branch` branch of the AMReX repo:

    > cd /path/to/mfix
    > git -C subprojects/amrex checkout my-amrex-branch
    > git status
    ...
    modified:   subprojects/amrex (new commits)

The branch `my-amrex-branch` will then be used when building MFIX-Exa.

To revert to the default version of the AMReX submodule, run `git submodule update`:

    > cd /path/to/mfix
    > git submodule update

You can edit, commit, pull, and push AMReX changes from `subprojects/amrex`. AMReX development is outside the scope of this document. Run `git status` in the top-level MFix-Exa repo to see whether the AMReX submodule has new commits, modified files, or untracked files.

To update the AMReX submodule referenced by MFIX-Exa:

    > git -C subprojects/amrex checkout UPDATED_AMREX_COMMIT_SHA1
    > git add subprojects/amrex
    > git commit -m 'Updating AMReX version'

This will only update the AMReX SHA-1 referenced by MFIX-Exa. Uncommitted AMReX changes and untracked AMReX files under `subprojects/amrex` are not added by `git add subprojects/amrex`. (To commit to the AMReX repo, change directories to `subprojects/amrex` and run Git commands there, before `git add subprojects/amrex`.)

Note

Only update the AMReX submodule reference in coordination with the other MFIX-Exa developers!

### STANDALONE instructions

#### Building AMReX

Clone AMReX from the official Git repository. Note that the only branch available is development:

    > git clone https://github.com/AMReX-Codes/amrex.git

Next, configure, build and install AMReX as follows:

    > cd amrex
    > mkdir build
    > cd build
    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DAMReX_PARTICLES=yes -DAMReX_EB=yes -DAMReX_PLOTFILE_TOOLS=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute_path_to_amrex_installdir ..
    > make install

The options AMReX_PARTICLES=yes, AMReX_EB=yes and AMReX_PLOTFILE_TOOLS=yes are required by MFIX-Exa. `[other amrex options]` in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the AMReX user guide for more details on building AMReX with CMake.

#### Building MFIX-Exa

Clone and build MFIX-Exa:

    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    > mkdir build
    > cd build
    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
    > make -j

Passing `-DAMReX_ROOT=/absolute/path/to/amrex/installdir` instructs CMake to search `/absolute/path/to/amrex/installdir` before searching system paths for an available AMReX installation. `AMReX_ROOT` can also be set as an environmental variable instead of passing it as a command line option. `[mfix options]` indicates any of the configuration option listed in the table below.

Table 1 MFIX-Exa configuration options
Option name | Description | Possible values | Default value
--- | --- | --- | ---
CMAKE_CXX_FLAGS | User-defined C++ flags | valid C++ compiler flags | None
CMAKE_CUDA_FLAGS | User-defined CUDA flags | valid CUDA compiler flags | None
MFIX_MPI | Enable build with MPI | no/yes | yes
MFIX_OMP | Enable build with OpenMP | no/yes | no
MFIX_GPU_ BACKEND | On-node, accelerated GPU backend | NONE,SYSCL,CUDA,HIP | NONE
MFIX_HYPRE | Enable HYPRE support | no/yes | no
MFIX_FPE | Build with Floating-Point Exceptions checks | no/yes | no
MFIX_CSG | Build with CSG support | no/yes | no
MFIX_MPI_ THREAD_MULTIPLE | Concurrent MPI calls from multiple threads | no/yes | no

### Few more notes on building MFIX-Exa with cmake

The system defaults compilers can be overwritten as follows:

    > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> [options]  ..

When building on a platform that uses the `module` utility, use either the above command (with full path to the compilers) or the following:

    > cmake -DCMAKE_CXX_COMPILER=CC [options] ..

MFIX-Exa uses the same compiler flags used to build AMReX, unless `CMAKE_CXX_FLAGS` is explicitly provided, or the environmental variable `CXXFLAGS` is set.

For GPU builds, MFIX-Exa relies on the AMReX GPU build infrastructure . The target architecture to build for can be specified via the AMReX configuration option `-DAMReX_CUDA_ARCH=<target-architecture>`, or by defining the environmental variable `AMREX_CUDA_ARCH` (all caps). If no GPU architecture is specified, CMake will try to determine which GPU is supported by the system.