Update CMake doc

Building MFiX-Exa with CMake

============================

There are two options to building MFiX-Exa with cmake:

CMake build is a two-step process. First ``cmake`` is invoked to create

configuration files and makefiles in a chosen directory (``builddir``).

Next, the actual build is performed by invoking ``make`` from within ``builddir``.

The CMake build process is summarized as follows:

o **SUPERBUILD (default):** Utilities download and build AMReX as part

of the MFiX-Exa build process. This method is strongly encouraged as it

ensures configure options for MFiX-Exa and AMReX are consistent.

.. highlight:: console

::

    mkdir /path/to/builddir

    cd    /path/to/builddir

    cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix

    make  

In the above snippet, ``[options]`` indicates one or more options for the

customization of the build, as described later in this section.

If the option ``CMAKE_BUILD_TYPE`` is omitted,

``CMAKE_BUILD_TYPE=Release`` is assumed. 

There are two modes to build MFiX-Exa with cmake:

o **Using an existing AMReX Library:** MFiX-Exa is linked to an existing

o **STANDALONE:** MFiX-Exa source code is built separately and linked to an existing

AMReX installation. This is ideal for continuous integration severs (CI)

and regression testing applications. AMReX library version must meet

MFiX-Exa requirements.

and regression testing applications. AMReX library version and configuration options

must meet MFiX-Exa requirements.

SUPERBUILD Instructions (recommended)

-------------------------------------

o **SUPERBUILD (recommended):** AMReX is downloaded and built as part

of the MFiX-Exa build process. This method is strongly encouraged as it

ensures that the configuration options for MFiX-Exa and AMReX are consistent.

**STANDALONE** instructions

---------------------------------------------------------------------

Build AMReX Library

~~~~~~~~~~~~~~~~~~~

When building in **SUPERBUILD** mode, MFiX-Exa build system will take

care of downloading, configuring and installing AMReX as part of the

MFiX-Exa build. The following commands build MFiX-Exa and AMReX in a

single step.

Clone AMReX from the official Git repository and checkout the

*development* branch:

.. code:: shell

    > git clone https://github.com/AMReX-Codes/amrex.git

    > cd amrex

    > git checkout development

Next, configure, build and install AMReX as follows:

.. code:: shell

    > mkdir build

    > cd build

    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DENABLE_PARTICLES=yes -DENABLE_AMRDATA=yes -DENABLE_EB=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir ..

    > make install

The options **ENABLE\_PARTICLES=yes**, **ENABLE\_AMRDATA=yes**, and **ENABLE\_EB=yes** are required by MFiX-Exa. ``[other amrex options]`` in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_ for more details on building AMReX with CMake.    

Building MFiX-Exa

~~~~~~~~~~~~~~~~~

Clone and build MFiX-Exa:

.. code:: shell

    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

    > cd mfix

    > mkdir build

    > cd build

    > cmake CONFIG_OPTIONS ..

    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..

    > make -j

AMReX is built the first time the ``make`` command is issued. No

external installation of AMReX is needed. However, internet access to

the AMReX github repository is required. The optional string

``CONFIG_OPTIONS`` allows to customize the build of both AMReX and

MFiX-Exa. ``CONFIG_OPTIONS`` is a list of one or more configuration

options given in the form **-D**\ *OPTION=VALUE*\ \`.

The table below lists configuration options, possible values, and their

effect on the build. Options prefixed by ``AMREX_`` are specific to the

build of AMReX.

Passing ``-DAMReX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to search

``/absolute/path/to/amrex/installdir`` before searching system paths

for an available AMReX installation.

``AMReX_ROOT`` can also be set as an environmental variable instead of passing it as a command line option.

``[mfix options]`` indicates any of the configuration option listed in the table below.

+-----------------+------------------------------+------------------+-------------+

| Option name     | Description                  | Possible values  | Default     |

|                 |                              |                  | value       |

+=================+==============================+==================+=============+

| DEBUG           | Build in debug mode          | ON/OFF           | OFF         |

| CMAKE\_Fortran\ | User-defined Fortran flags   | valid Fortran    | None        |

| _FLAGS          |                              | compiler flags   |             |

+-----------------+------------------------------+------------------+-------------+

| CMAKE\_Fortran\ | User-defined Fortran flags   | valid F90        | None        |

| _FLAGS          | for MFiX build               | compiler flags   |             |

| CMAKE\_CXX\     | User-defined C++ flags       | valid C++        | None        |

| _FLAGS          |                              | compiler flags   |             |

+-----------------+------------------------------+------------------+-------------+

| CMAKE\_CXX\_FLA | User-defined C++ flags for   | valid C++        | None        |

| GS              | MFiX build                   | compiler flags   |             |

| CMAKE\_CUDA\    | User-defined CUDA flags      | valid CUDA       | None        |

| _FLAGS          |                              | compiler flags   |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_Fortran\ | User-defined Fortran flags   | valid F90        | None        |

| _FLAGS          | for AMReX build              | compiler flags   |             |

| ENABLE\_MPI     | Enable build with MPI        | no/yes           | yes         |

|                 |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_CXX\_FLA | User-defined C++ flags for   | valid C++        | None        |

| GS              | AMReX build                  | compiler flags   |             |

| ENABLE\_OMP     | Enable build with OpenMP     | no/yes           | no          |

|                 |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| ENABLE\_MPI     | Enable build with MPI        | 0/1              | 1           |

| ENABLE\_CUDA    | Enable build with CUDA       | no/yes           | no          |

|                 |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| ENABLE\_OMP     | Enable build with OpenMP     | 0/1              | 0           |

| ENABLE\_HYPRE   | Enable HYPRE support         | no/yes           | no          |

|                 |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| ENABLE\_FPE     | Build with Floating-Point    | 0/1              | 0           |

| ENABLE\_FPE     | Build with Floating-Point    | no/yes           | no          |

|                 | Exceptions checks            |                  |             |

+-----------------+------------------------------+------------------+-------------+

| ENABLE\_PTESTS  | Include tests for projection | 0/1              | 0           |

|                 | method in Ctest suite        |                  |             |

+-----------------+------------------------------+------------------+-------------+

| ENABLE\_STESTS  | Include tests for SIMPLE     | 0/1              | 1           |

|                 | method in Ctest suite        |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Enable build with MPI        | 0/1              | 1           |

| MPI             |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Enable build with OpenMP     | 0/1              | 0           |

| OMP             |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Enable double precision      | 0/1              | 1           |

| DP              |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Enable double precision in   | 0/1              | 1           |

| DP\_PARTICLES   | particles classes            |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include profiling info       | 0/1              | 0           |

| BASE\_PROFILE   |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include tiny profiling info  | 0/1              | 0           |

| TINY\_PROFILE   |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include communicators        | 0/1              | 0           |

| COMM\_PROFILE   | profiling info               |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include trace profiling info | 0/1              | 0           |

| TRACE\_PROFILE  |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include memory profiling     | 0/1              | 0           |

| MEM\_PROFILE    | info                         |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include backtrace info       | 0/1              | 0           |

| BACKTRACE       |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Include profile parser       | 0/1              | 0           |

| PROFPARSER      |                              |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Build position-independent   | 0/1              | 0           |

| PIC             | code                         |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_ENABLE\_ | Build position-independent   | 0/1              | 0           |

| ASSERTION       | code                         |                  |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_GIT\_COM | AMReX commit to be used in   | valid git commit | None        |

| MIT             | the build                    | id/branch        |             |

+-----------------+------------------------------+------------------+-------------+

| AMREX\_INSTALL\ | Global path to AMReX install | valid global     | None        |

| _DIR            | directory                    | path             | (superbuild |

|                 |                              |                  | )           |

+-----------------+------------------------------+------------------+-------------+

``SUPERBUILD mode is enabled automatically when AMREX_INSTALL_DIR is not given.``

Example: build mfix with custom fortran flags, AMReX profiling enabled

and single precision particles:

::

    cmake -DMFiX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 ..

Building MFiX-Exa using a separate AMReX installation (no superbuild)

---------------------------------------------------------------------

Build AMReX Library

~~~~~~~~~~~~~~~~~~~

Clone AMReX from the official Git repository and checkout the

*development* branch.

.. code:: shell

    > git clone https://github.com/AMReX-Codes/amrex.git

    > cd amrex

    > git checkout development

Next, configure, build and install AMReX as follows:

.. code:: shell

    > cmake AMREX_CONFIG_OPTIONS -DENABLE_PARTICLES=1 -DENABLE_AMRDATA=1 -DENABLE_EB=1 -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir .

    > make install

SUPERBUILD Instructions (recommended)

-------------------------------------

Here,\ ``AMREX_CONFIG_OPTIONS`` are optional configure options for

AMReX. Please refer to the AMReX user guide for a list of all the

possible configuration options. The only options required are

**-DENABLE\_PARTICLES=1**, **-DENABLE\_AMRDATA=1**, and

**-DENABLE\_EB=1**.

If no AMReX installation is found on the system, or if one is found but does not meet MFiX-Exa requirements, MFiX-Exa CMake falls back to **SUPERBUILD** mode.

When building in **SUPERBUILD** mode, the AMReX git repo is checked out via a git submodule before AMReX CMake build system is included into MFiX-Exa CMake infrastructure. Consequently, MFiX-Exa CMake inherents AMReX's CMake targets and configuration options, that is, MFiX-Exa and AMReX are configured and built as a single entity.   

Building MFiX-Exa

~~~~~~~~~~~~~~~~~

Clone and build MFiX-Exa.

Assuming no valid AMReX installation is present on the target system, and ``AMReX_ROOT`` is not set in the environment, the following code will build  MFiX-Exa in **SUPERBUILD** mode:  

.. code:: shell

    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

    > cd mfix

    > mkdir build

    > cd build

    > cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir ..

    > cmake [amrex options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..

    > make -j

Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options,

that is, any option NOT prefixed by ``AMREX_`` in the table above.

Options prefixed by ``AMREX_`` are always ignored when using an external

AMReX installation.

``[amrex options]`` is a list of any of the AMReX configuration options listed in the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_

Few more notes on building MFiX-Exa

-----------------------------------

.. code:: shell

    > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> CONFIG_OPTIONS  ..

    > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> [options]  ..

When building on a platform that uses the ``module`` utility, use either

the above command (with full path to the compilers) or the following:

.. code:: shell

    > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS  ..

    > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn [options] ..

MFiX-Exa uses the same compiler flags used to build AMReX, unless

``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or