From 11647c5d344f8d0d7e60a68d119eef624987e21e Mon Sep 17 00:00:00 2001
From: Mark Meredith <mark.meredith@netl.doe.gov>
Date: Tue, 18 Jun 2019 14:24:13 -0400
Subject: [PATCH] Rewrite CMake build instructions for submodules

---
 docs/source/BuildingCMake.rst | 342 +++++++++++++++++++++++-----------
 1 file changed, 230 insertions(+), 112 deletions(-)

diff --git a/docs/source/BuildingCMake.rst b/docs/source/BuildingCMake.rst
index c7f6920..9e886ae 100644
--- a/docs/source/BuildingCMake.rst
+++ b/docs/source/BuildingCMake.rst
@@ -1,153 +1,271 @@
 Building MFiX-Exa with CMake
 ============================
 
-CMake build is a two-step process. First ``cmake`` is invoked to create
-configuration files and makefiles in a chosen directory (``builddir``).
-Next, the actual build is performed by invoking ``make`` from within ``builddir``.
-The CMake build process is summarized as follows:
+To build MFiX-Exa, first get the source, run ``cmake`` to configure and generate
+Makefiles, and run ``make`` to build MFiX-Exa.
 
-.. highlight:: console
+Throughout this section, ``builddir`` will be used as the name of the directory
+used for building MFiX-Exa.
 
-::
 
-    mkdir /path/to/builddir
-    cd    /path/to/builddir
-    cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix
-    make  
+Getting the Source
+------------------
+
+Use Git to clone MFiX-Exa from the Gitlab repository. (Requires a user account
+for the NETL MFiX-Exa Gitlab server.)
+
+.. code:: shell
 
-In the above snippet, ``[options]`` indicates one or more options for the
-customization of the build, as described later in this section.
-If the option ``CMAKE_BUILD_TYPE`` is omitted,
-``CMAKE_BUILD_TYPE=Release`` is assumed. 
+    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
+    > cd mfix
 
-There are two modes to build MFiX-Exa with cmake:
+The rest of the build instructions assume that the current directory is
+``mfix``, the top directory of the MFiX-Exa repo.
 
-o **STANDALONE:** MFiX-Exa source code is built separately and linked to an existing
-AMReX installation. This is ideal for continuous integration severs (CI)
-and regression testing applications. AMReX library version and configuration options
-must meet MFiX-Exa requirements.
 
-o **SUPERBUILD (recommended):** AMReX is downloaded and built as part
-of the MFiX-Exa build process. This method is strongly encouraged as it
-ensures that the configuration options for MFiX-Exa and AMReX are consistent.
+Default AMReX (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+By default, AMReX is downloaded and built as part of the MFiX-Exa build process
+(**SUPERBUILD** mode). This method is recommended to ensure the
+configuration options for MFiX-Exa and AMReX are compatible.
 
-**STANDALONE** instructions
----------------------------------------------------------------------
+Proceed to :ref:`Configuring`.
 
-Build AMReX Library
-~~~~~~~~~~~~~~~~~~~
+.. note::
 
-Clone AMReX from the official Git repository and checkout the
-*development* branch:
+      When running ``cmake``, ``subprojects/amrex`` will be cloned from
+      https://github.com/AMReX-Codes/amrex if it does not already exist. You can
+      also clone it manually if you want:
 
-.. code:: shell
+      .. code:: shell
 
-    > git clone https://github.com/AMReX-Codes/amrex.git
-    > cd amrex
-    > git checkout development
+          > git submodule update --init --recursive  #  OPTIONAL: cmake will do this automatically
 
-Next, configure, build and install AMReX as follows:
+Custom AMReX
+~~~~~~~~~~~~
 
-.. code:: shell
+For MFiX-Exa development or testing, you may need to build with a different
+branch or release of AMReX.
 
-    > mkdir build
-    > cd build
-    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DENABLE_PARTICLES=yes -DENABLE_AMRDATA=yes -DENABLE_EB=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir ..
-    > make install
+The ``subprojects/amrex`` directory is a Git repo, so use all standard Git
+commands. Either ``cd subprojects/amrex`` to run Git commands in the ``amrex``
+directory, or use ``git -C subprojects/amrex`` in the MFiX-Exa repo. For
+instance, to build with the ``my-amrex-branch`` branch of the AMReX repo:
 
-The options **ENABLE\_PARTICLES=yes**, **ENABLE\_AMRDATA=yes**, and **ENABLE\_EB=yes** are required by MFiX-Exa. ``[other amrex options]`` in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_ for more details on building AMReX with CMake.    
+.. code:: shell
 
+    > git -C subprojects/amrex checkout my-amrex-branch
+    > git status
+    ...
+    modified:   subprojects/amrex (new commits)
 
-Building MFiX-Exa
-~~~~~~~~~~~~~~~~~
+The ``my-amrex-branch`` will then be used when building MFiX-Exa.
 
-Clone and build MFiX-Exa:
+To revert to the default version of the AMReX submodule, run ``git submodule
+update``:
 
 .. code:: shell
 
-    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
-    > mkdir build
-    > cd build
-    > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir ..
-    > make -j
-
-    
-Passing ``-DAMReX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to search
-``/absolute/path/to/amrex/installdir`` before searching system paths
-for an available AMReX installation.
-``AMReX_ROOT`` can also be set as an environmental variable instead of passing it as a command line option.
-
-
-``[mfix options]`` indicates any of the configuration option listed in the table below.
-
-+-----------------+------------------------------+------------------+-------------+
-| Option name     | Description                  | Possible values  | Default     |
-|                 |                              |                  | value       |
-+=================+==============================+==================+=============+
-| CMAKE\_Fortran\ | User-defined Fortran flags   | valid Fortran    | None        |
-| _FLAGS          |                              | compiler flags   |             |
-+-----------------+------------------------------+------------------+-------------+
-| CMAKE\_CXX\     | User-defined C++ flags       | valid C++        | None        |
-| _FLAGS          |                              | compiler flags   |             |
-+-----------------+------------------------------+------------------+-------------+
-| CMAKE\_CUDA\    | User-defined CUDA flags      | valid CUDA       | None        |
-| _FLAGS          |                              | compiler flags   |             |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_MPI     | Enable build with MPI        | no/yes           | yes         |
-|                 |                              |                  |             |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_OMP     | Enable build with OpenMP     | no/yes           | no          |
-|                 |                              |                  |             |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_CUDA    | Enable build with CUDA       | no/yes           | no          |
-|                 |                              |                  |             |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_HYPRE   | Enable HYPRE support         | no/yes           | no          |
-|                 |                              |                  |             |
-+-----------------+------------------------------+------------------+-------------+
-| ENABLE\_FPE     | Build with Floating-Point    | no/yes           | no          |
-|                 | Exceptions checks            |                  |             |
-+-----------------+------------------------------+------------------+-------------+
-
-
-SUPERBUILD Instructions (recommended)
--------------------------------------
-
-If no AMReX installation is found on the system, or if one is found but does not meet MFiX-Exa requirements, MFiX-Exa CMake falls back to **SUPERBUILD** mode.
-When building in **SUPERBUILD** mode, the AMReX git repo is checked out via a git submodule before AMReX CMake build system is included into MFiX-Exa CMake infrastructure. Consequently, MFiX-Exa CMake inherents AMReX's CMake targets and configuration options, that is, MFiX-Exa and AMReX are configured and built as a single entity.   
-
-
-Assuming no valid AMReX installation is present on the target system, and ``AMReX_ROOT`` is not set in the environment, the following code will build  MFiX-Exa in **SUPERBUILD** mode:  
+    > git submodule update
+    > git status
+    ...
+    nothing to commit, working tree clean
+
+You can edit, commit, pull, and push AMReX changes from ``subprojects/amrex``.
+AMReX development is outside the scope of this document. Run ``git status`` in
+the top-level MFix-Exa repo to see whether the AMReX submodule has new commits,
+modified files, or untracked files.
+
+To update the AMReX submodule reference by MFiX-Exa:
 
 .. code:: shell
 
-    > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
-    > cd mfix
-    > mkdir build
-    > cd build
-    > cmake [amrex options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] ..
-    > make -j
+    > git add subprojects/amrex
+    > git commit -m 'Updating AMReX version'
+
+This will only update the AMReX SHA-1 referenced by MFiX-Exa. Uncommitted AMReX
+changes and untracked AMReX files under ``subprojects/amrex`` are not added by
+``git add subprojects/amrex``. Change directories to ``subprojects/amrex`` and
+commit any AMReX changes first before ``git add subprojects/amrex``.
+
+.. note::
+
+    Only update the AMReX submodule reference in coordination with the other
+    MFiX-Exa developers!
+
+Proceed to :ref:`Configuring`.
+
+
+.. _external_amrex:
+
+External AMReX
+~~~~~~~~~~~~~~
+
+It is also possible to build MFiX-Exa with an existing AMReX installation.
+
+For instructions on building AMReX with CMake, refer to the `AMReX user guide
+<https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_
+
+When building AMReX, the options ``ENABLE\_PARTICLES=yes``,
+``ENABLE\_AMRDATA=yes``, and ``ENABLE\_EB=yes`` are required by MFiX-Exa.
+
+.. highlight:: console
+
+::
+
+    > cd /path/to/amrex/source
+    > cmake -S . -B build \
+          -DENABLE_PARTICLES=yes \
+          -DENABLE_AMRDATA=yes \
+          -DENABLE_EB=yes \
+          [other amrex options] \
+          -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir
+    > make -C build install
+
+If AMReX is built with the option:
+
+.. highlight:: console
+
+::
+
+    -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/amrex/installdir
+
+Then MFiX-Exa should set ``AMREX_ROOT`` to that directory when running
+``cmake``:
+
+.. highlight:: console
 
-``[amrex options]`` is a list of any of the AMReX configuration options listed in the `AMReX user guide <https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_
+::
+
+    -DAMREX_ROOT=/absolute/path/to/amrex/installdir
+
+Proceed to :ref:`Configuring`.
+
+
+.. _Configuring:
+
+Configuring and Generating Makefiles
+------------------------------------
+
+Run ``cmake`` to generate Makefiles for building MFiX-Exa:
+
+.. highlight:: console
+
+::
+
+    > cmake -S . -B builddir
+
+Proceed to :ref:`Building`.
+
+.. note::
+
+    By default MFiX-Exa is built with MPI support. Use ``-DENABLE_MPI=NO`` to
+    build without MPI.
 
 
-Few more notes on building MFiX-Exa
------------------------------------
+Specifying Options
+~~~~~~~~~~~~~~~~~~
 
-The system defaults compilers can be overwritten as follows:
+.. highlight:: console
+
+::
+
+    > cmake -S . -B builddir \
+        [amrex options] \
+        [mfix options]
+
+
+In the above snippet, ``[amrex options]`` is a list of any of the AMReX
+configuration options listed in the `AMReX User Guide.
+<https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html#building-with-cmake>`_
+``[mfix options]`` indicates any of the configuration option listed in the table
+below.
+
++-------------------------+----------------------------+--------------------------------+----------+
+| Option name             | Description                | Possible values                | Default  |
+|                         |                            |                                | value    |
++=========================+============================+================================+==========+
+| ``CMAKE_BUILD_TYPE``    | Debug vs Release build     |  Debug, Release, RelWithDebInfo| Release  |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``CMAKE_Fortran_FLAGS`` | User-defined Fortran flags | See your compiler manual       | None     |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``CMAKE_CXX_FLAGS``     | User-defined C++ flags     | See your compiler manual       | None     |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``CMAKE_CUDA_FLAGS``    | User-defined CUDA flags    | See your compiler manual       | None     |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``ENABLE_MPI``          | Enable MPI support         | no/yes                         | yes      |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``ENABLE_OMP``          | Enable OpenMP support      | no/yes                         | no       |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``ENABLE_CUDA``         | Enable CUDA support        | no/yes                         | no       |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``ENABLE_HYPRE``        | Enable HYPRE support       | no/yes                         | no       |
++-------------------------+----------------------------+--------------------------------+----------+
+| ``ENABLE_FPE``          | Build with Floating-Point  | no/yes                         | no       |
+|                         | Exceptions checks          |                                |          |
++-------------------------+----------------------------+--------------------------------+----------+
+
+
+Specifying Compilers
+~~~~~~~~~~~~~~~~~~~~
+
+The default system default compilers can be overwritten as follows:
 
 .. code:: shell
 
-    > cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> [options]  ..
+    > cmake -S . -B builddir \
+          -DCMAKE_CXX_COMPILER=<c++-compiler> \
+          -DCMAKE_Fortran_COMPILER=<f90-compiler>
+
+.. note::
 
-When building on a platform that uses the ``module`` utility, use either
-the above command (with full path to the compilers) or the following:
+    If ``ENABLE_MPI`` is set (the default), then ``CMAKE_CXX_COMPILER`` should
+    be a compiler wrapper such as ``mpic++``, or there will likely be build
+    errors. Likewise, ``CMAKE_Fortran_COMPILER`` should be set to ``mpifort``
+    (or whatever is your MPI Fortran wrapper).
+
+
+Specifying external AMReX
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If building with :ref:`external_amrex`, specify the path to AMReX with
+``-DAMREX_ROOT``:
+
+.. highlight:: console
+
+::
+
+    > cmake -S . -B builddir \
+        -DAMREX_ROOT=/absolute/path/to/amrex/installdir
+
+Passing ``-DAMREX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to
+search ``/absolute/path/to/amrex/installdir``, and then search system paths, for
+an available AMReX installation.
+
+Alternatively, set the ``AMREX_ROOT`` environmental variable instead of passing
+it as a command line option.
+
+.. note::
+
+   MFiX-Exa needs to be built with the same CMake options used to build the
+   external AMReX installation. This is another reason when the default
+   (Superbuild) option is recommended.
+
+
+.. _Building:
+
+Building
+--------
+
+To build MFiX-Exa:
 
 .. code:: shell
 
-    > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn [options] ..
+    > make -C builddir
+
+To build all test cases:
+
+.. code:: shell
 
-MFiX-Exa uses the same compiler flags used to build AMReX, unless
-``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or
-the environmental variables ``FFLAGS``/``CXXFLAGS`` are set.
+    > make -C builddir build_tests
-- 
GitLab