diff --git a/docs/source/BuildingCMake.rst b/docs/source/BuildingCMake.rst
index bdcae0b8a84df23196cf134b39bf92853fdf9fce..a881da6e8486931ad40efb6d93d5d86d7d1101d2 100644
--- a/docs/source/BuildingCMake.rst
+++ b/docs/source/BuildingCMake.rst
@@ -1,24 +1,49 @@
Building MFiX-Exa with CMake
============================
-There are two options to building MFiX-Exa with cmake:
+CMake build is a two-step process. First ``cmake`` is invoked to create
+configuration files and makefiles in a chosen directory (``builddir``).
+Next, the actual build is performed by invoking ``make`` from within ``builddir``.
+The CMake build process is summarized as follows:
-o **SUPERBUILD (default):** Utilities download and build AMReX as part
+.. highlight:: console
+
+::
+
+ mkdir /path/to/builddir
+ cd /path/to/builddir
+ cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix
+ make
+
+In the above snippet, ``[options]`` indicates one or more options for the
+customization of the build, as described later in this section.
+If the option ``CMAKE_BUILD_TYPE`` is omitted,
+``CMAKE_BUILD_TYPE=Release`` is assumed.
+
+There are two modes to build MFiX-Exa with cmake:
+
+o **SUPERBUILD (recommended):** AMReX is downloaded and built as part
of the MFiX-Exa build process. This method is strongly encouraged as it
-ensures configure options for MFiX-Exa and AMReX are consistent.
+ensures that the configuration options for MFiX-Exa and AMReX are consistent.
-o **Using an existing AMReX Library:** MFiX-Exa is linked to an existing
+o **STANDALONE:** MFiX-Exa source code is built separately and linked to an existing
AMReX installation. This is ideal for continuous integration severs (CI)
-and regression testing applications. AMReX library version must meet
-MFiX-Exa requirements.
+and regression testing applications. AMReX library version and configuration options
+must meet MFiX-Exa requirements.
+
+
+
+.. note::
+ **MFiX-Exa requires CMake 3.14 or higher.**
+
SUPERBUILD Instructions (recommended)
-------------------------------------
-When building in **SUPERBUILD** mode, MFiX-Exa build system will take
-care of downloading, configuring and installing AMReX as part of the
-MFiX-Exa build. The following commands build MFiX-Exa and AMReX in a
-single step.
+By default MFiX-Exa CMake looks for an existing AMReX installation on the system. If none is found, it falls back to **SUPERBUILD** mode.
+In this mode, MFiX-Exa CMake inherents AMReX CMake targets and configuration options, that is, MFiX-Exa and AMReX are configured and built as a single entity
+
+Assuming no valid AMReX installation is present on the target system, and ``AMReX_ROOT`` is not set (see :ref:`sec:build:standalone`), the following code will build MFiX-Exa in **SUPERBUILD** mode:
.. code:: shell
@@ -26,117 +51,83 @@ single step.
> cd mfix
> mkdir build
> cd build
- > cmake CONFIG_OPTIONS ..
+ > cmake [amrex options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..
> make -j
-AMReX is built the first time the ``make`` command is issued. No
-external installation of AMReX is needed. However, internet access to
-the AMReX github repository is required. The optional string
-``CONFIG_OPTIONS`` allows to customize the build of both AMReX and
-MFiX-Exa. ``CONFIG_OPTIONS`` is a list of one or more configuration
-options given in the form **-D**\ *OPTION=VALUE*\ \`.
+``[amrex options]`` in the snippet above is a list of any of the AMReX configuration options listed in the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_
-The table below lists configuration options, possible values, and their
-effect on the build. Options prefixed by ``AMREX_`` are specific to the
-build of AMReX.
-+-----------------+------------------------------+------------------+-------------+
-| Option name | Description | Possible values | Default |
-| | | | value |
-+=================+==============================+==================+=============+
-| DEBUG | Build in debug mode | ON/OFF | OFF |
-+-----------------+------------------------------+------------------+-------------+
-| CMAKE\_Fortran\ | User-defined Fortran flags | valid F90 | None |
-| _FLAGS | for MFiX build | compiler flags | |
-+-----------------+------------------------------+------------------+-------------+
-| CMAKE\_CXX\_FLA | User-defined C++ flags for | valid C++ | None |
-| GS | MFiX build | compiler flags | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_Fortran\ | User-defined Fortran flags | valid F90 | None |
-| _FLAGS | for AMReX build | compiler flags | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_CXX\_FLA | User-defined C++ flags for | valid C++ | None |
-| GS | AMReX build | compiler flags | |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_MPI | Enable build with MPI | 0/1 | 1 |
-| | | | |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_OMP | Enable build with OpenMP | 0/1 | 0 |
-| | | | |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_FPE | Build with Floating-Point | 0/1 | 0 |
-| | Exceptions checks | | |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_PTESTS | Include tests for projection | 0/1 | 0 |
-| | method in Ctest suite | | |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_STESTS | Include tests for SIMPLE | 0/1 | 1 |
-| | method in Ctest suite | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Enable build with MPI | 0/1 | 1 |
-| MPI | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Enable build with OpenMP | 0/1 | 0 |
-| OMP | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Enable double precision | 0/1 | 1 |
-| DP | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Enable double precision in | 0/1 | 1 |
-| DP\_PARTICLES | particles classes | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include profiling info | 0/1 | 0 |
-| BASE\_PROFILE | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include tiny profiling info | 0/1 | 0 |
-| TINY\_PROFILE | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include communicators | 0/1 | 0 |
-| COMM\_PROFILE | profiling info | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include trace profiling info | 0/1 | 0 |
-| TRACE\_PROFILE | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include memory profiling | 0/1 | 0 |
-| MEM\_PROFILE | info | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include backtrace info | 0/1 | 0 |
-| BACKTRACE | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Include profile parser | 0/1 | 0 |
-| PROFPARSER | | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Build position-independent | 0/1 | 0 |
-| PIC | code | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_ENABLE\_ | Build position-independent | 0/1 | 0 |
-| ASSERTION | code | | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_GIT\_COM | AMReX commit to be used in | valid git commit | None |
-| MIT | the build | id/branch | |
-+-----------------+------------------------------+------------------+-------------+
-| AMREX\_INSTALL\ | Global path to AMReX install | valid global | None |
-| _DIR | directory | path | (superbuild |
-| | | | ) |
-+-----------------+------------------------------+------------------+-------------+
+Working with the AMReX submodule
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-``SUPERBUILD mode is enabled automatically when AMREX_INSTALL_DIR is not given.``
+**SUPERBUILD** mode relies on a git submodule to checkout the AMReX git repository.
+If the AMReX submodule is not initialized, **SUPERBUILD** mode will initialize it and checkout
+the AMReX commit the submodule is pointing to.
+Instead, if the AMReX submodule has already been manually initialized and a custom commit has been checked out,
+**SUPERBUILD** mode will use that commit. For MFiX-Exa development or testing, you may need to build with a different
+branch or release of AMReX.
-Example: build mfix with custom fortran flags, AMReX profiling enabled
-and single precision particles:
+The ``subprojects/amrex`` directory is a Git repo, so use all standard Git
+commands. Either ``cd subprojects/amrex`` to run Git commands in the ``amrex``
+directory, or use ``git -C subprojects/amrex`` in the MFiX-Exa repo. For
+instance, to build with the ``my-amrex-branch`` branch of the AMReX repo:
-::
+.. code:: shell
+
+ > git -C subprojects/amrex checkout my-amrex-branch
+ > git status
+ ...
+ modified: subprojects/amrex (new commits)
+
+The branch ``my-amrex-branch`` will then be used when building MFiX-Exa.
+
+To revert to the default version of the AMReX submodule, run ``git submodule
+update``:
+
+.. code:: shell
+
+ > git submodule update
+ > git status
+ ...
+ nothing to commit, working tree clean
+
+You can edit, commit, pull, and push AMReX changes from ``subprojects/amrex``.
+AMReX development is outside the scope of this document. Run ``git status`` in
+the top-level MFix-Exa repo to see whether the AMReX submodule has new commits,
+modified files, or untracked files.
+
+The CMake variables ``AMREX_GIT_COMMIT_DEVELOP`` and
+``AMREX_GIT_COMMIT_DEVELOP`` have been removed. Instead, to update the AMReX
+submodule referenced by MFiX-Exa:
+
+.. code:: shell
- cmake -DMFiX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 ..
+ > git -C subprojects/amrex checkout UPDATED_AMREX_COMMIT_SHA1
+ > git add subprojects/amrex
+ > git commit -m 'Updating AMReX version'
-Building MFiX-Exa using a separate AMReX installation (no superbuild)
+This will only update the AMReX SHA-1 referenced by MFiX-Exa. Uncommitted AMReX
+changes and untracked AMReX files under ``subprojects/amrex`` are not added by
+``git add subprojects/amrex``. (To commit to the AMReX repo, change directories
+to ``subprojects/amrex`` and run Git commands there, before ``git add
+subprojects/amrex``.)
+
+.. note::
+
+ Only update the AMReX submodule reference in coordination with the other
+ MFiX-Exa developers!
+
+
+.. _sec:build:standalone:
+
+**STANDALONE** instructions
---------------------------------------------------------------------
-Build AMReX Library
+Building AMReX
~~~~~~~~~~~~~~~~~~~
Clone AMReX from the official Git repository and checkout the
-*development* branch.
+*development* branch:
.. code:: shell
@@ -148,37 +139,67 @@ Next, configure, build and install AMReX as follows:
.. code:: shell
- > cmake . AMREX_CONFIG_OPTIONS \
- -DENABLE_PARTICLES=1 \
- -DENABLE_AMRDATA=1 \
- -DENABLE_EB=1 \
- -DENABLE_3D_NODAL_MLMG=1 \
- -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir
-
+ > mkdir build
+ > cd build
+ > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DENABLE_PARTICLES=yes -DENABLE_AMRDATA=yes -DENABLE_EB=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir ..
> make install
-Here,\ ``AMREX_CONFIG_OPTIONS`` are optional configure options for AMReX. Please
-refer to the AMReX user guide for a list of all the possible configuration
-options. The only options required are **-DENABLE\_PARTICLES=1**,
-**-DENABLE\_AMRDATA=1**, **-DENABLE\_EB=1**, and **-DENABLE_3D_NODAL_MLMG=1**.
+The options **ENABLE\_PARTICLES=yes**, **ENABLE\_AMRDATA=yes**, and **ENABLE\_EB=yes** are required by MFiX-Exa. ``[other amrex options]`` in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_ for more details on building AMReX with CMake.
+
Building MFiX-Exa
~~~~~~~~~~~~~~~~~
-Clone and build MFiX-Exa.
+Clone and build MFiX-Exa:
.. code:: shell
> git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
> mkdir build
> cd build
- > cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir ..
+ > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
> make -j
-Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options,
-that is, any option NOT prefixed by ``AMREX_`` in the table above.
-Options prefixed by ``AMREX_`` are always ignored when using an external
-AMReX installation.
+
+Passing ``-DAMReX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to search
+``/absolute/path/to/amrex/installdir`` before searching system paths
+for an available AMReX installation.
+``AMReX_ROOT`` can also be set as an environmental variable instead of passing it as a command line option.
+
+
+``[mfix options]`` indicates any of the configuration option listed in the table below.
+
++-----------------+------------------------------+------------------+-------------+
+| Option name | Description | Possible values | Default |
+| | | | value |
++=================+==============================+==================+=============+
+| CMAKE\_Fortran\ | User-defined Fortran flags | valid Fortran | None |
+| _FLAGS | | compiler flags | |
++-----------------+------------------------------+------------------+-------------+
+| CMAKE\_CXX\ | User-defined C++ flags | valid C++ | None |
+| _FLAGS | | compiler flags | |
++-----------------+------------------------------+------------------+-------------+
+| CMAKE\_CUDA\ | User-defined CUDA flags | valid CUDA | None |
+| _FLAGS | | compiler flags | |
++-----------------+------------------------------+------------------+-------------+
+| ENABLE\_MPI | Enable build with MPI | no/yes | yes |
+| | | | |
++-----------------+------------------------------+------------------+-------------+
+| ENABLE\_OMP | Enable build with OpenMP | no/yes | no |
+| | | | |
++-----------------+------------------------------+------------------+-------------+
+| ENABLE\_CUDA | Enable build with CUDA | no/yes | no |
+| | | | |
++-----------------+------------------------------+------------------+-------------+
+| ENABLE\_HYPRE | Enable HYPRE support | no/yes | no |
+| | | | |
++-----------------+------------------------------+------------------+-------------+
+| ENABLE\_FPE | Build with Floating-Point | no/yes | no |
+| | Exceptions checks | | |
++-----------------+------------------------------+------------------+-------------+
+
+
+
Few more notes on building MFiX-Exa
-----------------------------------
@@ -187,14 +208,14 @@ The system defaults compilers can be overwritten as follows:
.. code:: shell
- > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> CONFIG_OPTIONS ..
+ > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> [options] ..
When building on a platform that uses the ``module`` utility, use either
the above command (with full path to the compilers) or the following:
.. code:: shell
- > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS ..
+ > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn [options] ..
MFiX-Exa uses the same compiler flags used to build AMReX, unless
``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or
diff --git a/docs/source/InputsMonitors.rst b/docs/source/InputsMonitors.rst
index 3d54a4a1d20c83d2984c9b314323d715603c44d2..580cea30c22ff5b3804cdc70e4da91a6c26dcf1e 100644
--- a/docs/source/InputsMonitors.rst
+++ b/docs/source/InputsMonitors.rst
@@ -4,16 +4,16 @@ Monitors
========
The following inputs must be preceded by "amr" and control whether to compute
-monitors, i.e., spatial averages, and how often to output the results.
-n is the number of monitors implicitly defined by the size of avg_region_x_w.
+monitors, i.e., spatial averages, and how often to output the results.
+n is the number of monitors implicitly defined by the size of avg_region_x_w.
+------------------+-----------------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+==================+=======================================================================+=============+===========+
| avg_int | Interval, in number of CFD dt's, to write output | Int | -1 |
+------------------+-----------------------------------------------------------------------+-------------+-----------+
-| avg_file | Base file name which is appended with the data type (vel_p_, p_g_, | String | avg_region|
-| | ep_g_ or vel_g_), the number of this type of averaging, and the .csv | | |
+| avg_file | Base file name which is appended with the data type (vel_p, p_g, | String | avg_region|
+| | ep_g or vel_g), the number of this type of averaging, and the .csv | | |
| | file extension | | |
+------------------+-----------------------------------------------------------------------+-------------+-----------+
| avg_vel_p | Average and save particle velocity (if 1) | n*Int | 0 |
@@ -36,6 +36,3 @@ n is the number of monitors implicitly defined by the size of avg_region_x_w.
+------------------+-----------------------------------------------------------------------+-------------+-----------+
| avg_region_z_t | Upper bound of averaging region in z-direction | n*Real | None |
+------------------+-----------------------------------------------------------------------+-------------+-----------+
-
-
-
diff --git a/docs/source/InputsProblemDefinition.rst b/docs/source/InputsProblemDefinition.rst
index bd6c14c7f0dca0719e8e66913405d0b933efce85..66fef77f7509f2028d3723a9da8ef34640e100d7 100644
--- a/docs/source/InputsProblemDefinition.rst
+++ b/docs/source/InputsProblemDefinition.rst
@@ -27,12 +27,12 @@ The following inputs must be preceded by "geometry."
The following inputs must be preceded by "mfix."
-+----------------------+-------------------------------------------------------------------------+-------------+--------+
++----------------------+-------------------------------------------------------------------------+----------+-----------+
| | Description | Type | Default |
+======================+=========================================================================+==========+===========+
| geometry | Which type of EB geometry are we using? | String | |
+----------------------+-------------------------------------------------------------------------+----------+-----------+
-| levelset__refinement | Refinement factor of levelset resolution relative to level 0 resolution | Int | 1 !
+| levelset__refinement | Refinement factor of levelset resolution relative to level 0 resolution | Int | 1 |
+----------------------+-------------------------------------------------------------------------+----------+-----------+
Setting basic EB walls can be specified by inputs preceded by "xlo", "xhi", "ylo", "yhi", "zlo", and "zhi"
diff --git a/docs/source/LoadBalancing.rst b/docs/source/LoadBalancing.rst
index 7496e3934c3b7ea16081b9e1f1e97057350dc60d..7c685d8617a693bcd8fc432e2389aebbfedbfe7c 100644
--- a/docs/source/LoadBalancing.rst
+++ b/docs/source/LoadBalancing.rst
@@ -9,23 +9,23 @@
Load Balancing
--------------
-The process of load balancing is typically independent of the process of grid creation;
-the inputs to load balancing are a given set of grids with a set of weights
+The process of load balancing is typically independent of the process of grid creation;
+the inputs to load balancing are a given set of grids with a set of weights
assigned to each grid. (The exception to this is the KD-tree approach in which the
grid creation process is governed by trying to balance the work in each grid.)
-Single-level load balancing algorithms are sequentially applied to each AMR level independently,
-and the resulting distributions are mapped onto the ranks taking into account the weights
+Single-level load balancing algorithms are sequentially applied to each AMR level independently,
+and the resulting distributions are mapped onto the ranks taking into account the weights
already assigned to them (assign heaviest set of grids to the least loaded rank)
Options supported by AMReX include:
-- Knapsack: the default weight of a grid in the knapsack algorithm is the number of grid cells,
- but AMReX supports the option to pass an array of weights – one per grid – or alternatively
+- Knapsack: the default weight of a grid in the knapsack algorithm is the number of grid cells,
+ but AMReX supports the option to pass an array of weights – one per grid – or alternatively
to pass in a MultiFab of weights per cell which is used to compute the weight per grid
-- SFC: enumerate grids with a space-filling Z-morton curve, then partition the
+- SFC: enumerate grids with a space-filling Z-morton curve, then partition the
resulting ordering across ranks in a way that balances the load
- Round-robin: sort grids and assign them to ranks in round-robin fashion -- specifically
- FAB *i* is owned by CPU *i*%N where N is the total number of MPI ranks.
+ FAB ``i`` is owned by CPU ``i % N`` where N is the total number of MPI ranks.