diff --git a/docs/source/BuildingCMake.rst b/docs/source/BuildingCMake.rst index bdcae0b8a84df23196cf134b39bf92853fdf9fce..9cf187332ac0e33c2ab753c430a4421ebcb21a88 100644 --- a/docs/source/BuildingCMake.rst +++ b/docs/source/BuildingCMake.rst @@ -1,184 +1,193 @@ Building MFiX-Exa with CMake ============================ -There are two options to building MFiX-Exa with cmake: +CMake build is a two-step process. First ``cmake`` is invoked to create +configuration files and makefiles in a chosen directory (``builddir``). +Next, the actual build is performed by invoking ``make`` from within ``builddir``. +The CMake build process is summarized as follows: -o **SUPERBUILD (default):** Utilities download and build AMReX as part -of the MFiX-Exa build process. This method is strongly encouraged as it -ensures configure options for MFiX-Exa and AMReX are consistent. +.. highlight:: console + +:: + + mkdir /path/to/builddir + cd /path/to/builddir + cmake [options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] /path/to/mfix + make + +In the above snippet, ``[options]`` indicates one or more options for the +customization of the build, as described later in this section. +If the option ``CMAKE_BUILD_TYPE`` is omitted, +``CMAKE_BUILD_TYPE=Release`` is assumed. + +There are two modes to build MFiX-Exa with cmake: -o **Using an existing AMReX Library:** MFiX-Exa is linked to an existing +o **STANDALONE:** MFiX-Exa source code is built separately and linked to an existing AMReX installation. This is ideal for continuous integration severs (CI) -and regression testing applications. AMReX library version must meet -MFiX-Exa requirements. +and regression testing applications. AMReX library version and configuration options +must meet MFiX-Exa requirements. -SUPERBUILD Instructions (recommended) -------------------------------------- +o **SUPERBUILD (recommended):** AMReX is downloaded and built as part +of the MFiX-Exa build process. This method is strongly encouraged as it +ensures that the configuration options for MFiX-Exa and AMReX are consistent. + + +**STANDALONE** instructions +--------------------------------------------------------------------- + +Build AMReX Library +~~~~~~~~~~~~~~~~~~~ + +Clone AMReX from the official Git repository and checkout the +*development* branch: + +.. code:: shell + + > git clone https://github.com/AMReX-Codes/amrex.git + > cd amrex + > git checkout development -When building in **SUPERBUILD** mode, MFiX-Exa build system will take -care of downloading, configuring and installing AMReX as part of the -MFiX-Exa build. The following commands build MFiX-Exa and AMReX in a -single step. +Next, configure, build and install AMReX as follows: + +.. code:: shell + + > mkdir build + > cd build + > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] -DENABLE_PARTICLES=yes -DENABLE_AMRDATA=yes -DENABLE_EB=yes [other amrex options] -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir .. + > make install + +The options **ENABLE\_PARTICLES=yes**, **ENABLE\_AMRDATA=yes**, and **ENABLE\_EB=yes** are required by MFiX-Exa. ``[other amrex options]`` in the snippet above refers to any other AMReX configuration option in addition to the required ones. Please refer to the `AMReX user guide `_ for more details on building AMReX with CMake. + + +Building MFiX-Exa +~~~~~~~~~~~~~~~~~ + +Clone and build MFiX-Exa: .. code:: shell > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git - > cd mfix > mkdir build > cd build - > cmake CONFIG_OPTIONS .. + > cmake -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] [mfix options] -DAMReX_ROOT=/absolute/path/to/amrex/installdir .. > make -j -AMReX is built the first time the ``make`` command is issued. No -external installation of AMReX is needed. However, internet access to -the AMReX github repository is required. The optional string -``CONFIG_OPTIONS`` allows to customize the build of both AMReX and -MFiX-Exa. ``CONFIG_OPTIONS`` is a list of one or more configuration -options given in the form **-D**\ *OPTION=VALUE*\ \`. -The table below lists configuration options, possible values, and their -effect on the build. Options prefixed by ``AMREX_`` are specific to the -build of AMReX. +Passing ``-DAMReX_ROOT=/absolute/path/to/amrex/installdir`` instructs CMake to search +``/absolute/path/to/amrex/installdir`` before searching system paths +for an available AMReX installation. +``AMReX_ROOT`` can also be set as an environmental variable instead of passing it as a command line option. + + +``[mfix options]`` indicates any of the configuration option listed in the table below. +-----------------+------------------------------+------------------+-------------+ | Option name | Description | Possible values | Default | | | | | value | +=================+==============================+==================+=============+ -| DEBUG | Build in debug mode | ON/OFF | OFF | +| CMAKE\_Fortran\ | User-defined Fortran flags | valid Fortran | None | +| _FLAGS | | compiler flags | | +-----------------+------------------------------+------------------+-------------+ -| CMAKE\_Fortran\ | User-defined Fortran flags | valid F90 | None | -| _FLAGS | for MFiX build | compiler flags | | +| CMAKE\_CXX\ | User-defined C++ flags | valid C++ | None | +| _FLAGS | | compiler flags | | +-----------------+------------------------------+------------------+-------------+ -| CMAKE\_CXX\_FLA | User-defined C++ flags for | valid C++ | None | -| GS | MFiX build | compiler flags | | +| CMAKE\_CUDA\ | User-defined CUDA flags | valid CUDA | None | +| _FLAGS | | compiler flags | | +-----------------+------------------------------+------------------+-------------+ -| AMREX\_Fortran\ | User-defined Fortran flags | valid F90 | None | -| _FLAGS | for AMReX build | compiler flags | | +| ENABLE\_MPI | Enable build with MPI | no/yes | yes | +| | | | | +-----------------+------------------------------+------------------+-------------+ -| AMREX\_CXX\_FLA | User-defined C++ flags for | valid C++ | None | -| GS | AMReX build | compiler flags | | +| ENABLE\_OMP | Enable build with OpenMP | no/yes | no | +| | | | | +-----------------+------------------------------+------------------+-------------+ -| ENABLE\_MPI | Enable build with MPI | 0/1 | 1 | +| ENABLE\_CUDA | Enable build with CUDA | no/yes | no | | | | | | +-----------------+------------------------------+------------------+-------------+ -| ENABLE\_OMP | Enable build with OpenMP | 0/1 | 0 | +| ENABLE\_HYPRE | Enable HYPRE support | no/yes | no | | | | | | +-----------------+------------------------------+------------------+-------------+ -| ENABLE\_FPE | Build with Floating-Point | 0/1 | 0 | +| ENABLE\_FPE | Build with Floating-Point | no/yes | no | | | Exceptions checks | | | +-----------------+------------------------------+------------------+-------------+ -| ENABLE\_PTESTS | Include tests for projection | 0/1 | 0 | -| | method in Ctest suite | | | -+-----------------+------------------------------+------------------+-------------+ -| ENABLE\_STESTS | Include tests for SIMPLE | 0/1 | 1 | -| | method in Ctest suite | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Enable build with MPI | 0/1 | 1 | -| MPI | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Enable build with OpenMP | 0/1 | 0 | -| OMP | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Enable double precision | 0/1 | 1 | -| DP | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Enable double precision in | 0/1 | 1 | -| DP\_PARTICLES | particles classes | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include profiling info | 0/1 | 0 | -| BASE\_PROFILE | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include tiny profiling info | 0/1 | 0 | -| TINY\_PROFILE | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include communicators | 0/1 | 0 | -| COMM\_PROFILE | profiling info | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include trace profiling info | 0/1 | 0 | -| TRACE\_PROFILE | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include memory profiling | 0/1 | 0 | -| MEM\_PROFILE | info | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include backtrace info | 0/1 | 0 | -| BACKTRACE | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Include profile parser | 0/1 | 0 | -| PROFPARSER | | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Build position-independent | 0/1 | 0 | -| PIC | code | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_ENABLE\_ | Build position-independent | 0/1 | 0 | -| ASSERTION | code | | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_GIT\_COM | AMReX commit to be used in | valid git commit | None | -| MIT | the build | id/branch | | -+-----------------+------------------------------+------------------+-------------+ -| AMREX\_INSTALL\ | Global path to AMReX install | valid global | None | -| _DIR | directory | path | (superbuild | -| | | | ) | -+-----------------+------------------------------+------------------+-------------+ -``SUPERBUILD mode is enabled automatically when AMREX_INSTALL_DIR is not given.`` -Example: build mfix with custom fortran flags, AMReX profiling enabled -and single precision particles: +SUPERBUILD Instructions (recommended) +------------------------------------- -:: +If no AMReX installation is found on the system, or if one is found but does not meet MFiX-Exa requirements, MFiX-Exa CMake falls back to **SUPERBUILD** mode. +When building in **SUPERBUILD** mode, the AMReX git repo is checked out via a git submodule before AMReX CMake build system is included into MFiX-Exa CMake infrastructure. Consequently, MFiX-Exa CMake inherents AMReX's CMake targets and configuration options, that is, MFiX-Exa and AMReX are configured and built as a single entity. - cmake -DMFiX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 .. -Building MFiX-Exa using a separate AMReX installation (no superbuild) ---------------------------------------------------------------------- +Assuming no valid AMReX installation is present on the target system, and ``AMReX_ROOT`` is not set in the environment, the following code will build MFiX-Exa in **SUPERBUILD** mode: -Build AMReX Library -~~~~~~~~~~~~~~~~~~~ +.. code:: shell -Clone AMReX from the official Git repository and checkout the -*development* branch. + > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git + > cd mfix + > mkdir build + > cd build + > cmake [amrex options] -DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel] .. + > make -j -.. code:: shell +``[amrex options]`` is a list of any of the AMReX configuration options listed in the `AMReX user guide `_ - > git clone https://github.com/AMReX-Codes/amrex.git - > cd amrex - > git checkout development -Next, configure, build and install AMReX as follows: +Working with the AMReX submodule +-------------------------------- + +For MFiX-Exa development or testing, you may need to build with a different +branch or release of AMReX. + +The ``subprojects/amrex`` directory is a Git repo, so use all standard Git +commands. Either ``cd subprojects/amrex`` to run Git commands in the ``amrex`` +directory, or use ``git -C subprojects/amrex`` in the MFiX-Exa repo. For +instance, to build with the ``my-amrex-branch`` branch of the AMReX repo: .. code:: shell - > cmake . AMREX_CONFIG_OPTIONS \ - -DENABLE_PARTICLES=1 \ - -DENABLE_AMRDATA=1 \ - -DENABLE_EB=1 \ - -DENABLE_3D_NODAL_MLMG=1 \ - -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir + > git -C subprojects/amrex checkout my-amrex-branch + > git status + ... + modified: subprojects/amrex (new commits) - > make install +The branch ``my-amrex-branch`` will then be used when building MFiX-Exa. -Here,\ ``AMREX_CONFIG_OPTIONS`` are optional configure options for AMReX. Please -refer to the AMReX user guide for a list of all the possible configuration -options. The only options required are **-DENABLE\_PARTICLES=1**, -**-DENABLE\_AMRDATA=1**, **-DENABLE\_EB=1**, and **-DENABLE_3D_NODAL_MLMG=1**. +To revert to the default version of the AMReX submodule, run ``git submodule +update``: -Building MFiX-Exa -~~~~~~~~~~~~~~~~~ +.. code:: shell + + > git submodule update + > git status + ... + nothing to commit, working tree clean -Clone and build MFiX-Exa. +You can edit, commit, pull, and push AMReX changes from ``subprojects/amrex``. +AMReX development is outside the scope of this document. Run ``git status`` in +the top-level MFix-Exa repo to see whether the AMReX submodule has new commits, +modified files, or untracked files. + +The CMake variables ``AMREX_GIT_COMMIT_DEVELOP`` and +``AMREX_GIT_COMMIT_DEVELOP`` have been removed. Instead, to update the AMReX +submodule referenced by MFiX-Exa: .. code:: shell - > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git - > mkdir build - > cd build - > cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir .. - > make -j + > git -C subprojects/amrex checkout UPDATED_AMREX_COMMIT_SHA1 + > git add subprojects/amrex + > git commit -m 'Updating AMReX version' + +This will only update the AMReX SHA-1 referenced by MFiX-Exa. Uncommitted AMReX +changes and untracked AMReX files under ``subprojects/amrex`` are not added by +``git add subprojects/amrex``. (To commit to the AMReX repo, change directories +to ``subprojects/amrex`` and run Git commands there, before ``git add +subprojects/amrex``.) + +.. note:: + + Only update the AMReX submodule reference in coordination with the other + MFiX-Exa developers! -Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options, -that is, any option NOT prefixed by ``AMREX_`` in the table above. -Options prefixed by ``AMREX_`` are always ignored when using an external -AMReX installation. Few more notes on building MFiX-Exa ----------------------------------- @@ -187,14 +196,14 @@ The system defaults compilers can be overwritten as follows: .. code:: shell - > cmake -DCMAKE_CXX_COMPILER= -DCMAKE_Fortran_COMPILER= CONFIG_OPTIONS .. + > cmake -DCMAKE_CXX_COMPILER= -DCMAKE_Fortran_COMPILER= [options] .. When building on a platform that uses the ``module`` utility, use either the above command (with full path to the compilers) or the following: .. code:: shell - > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS .. + > cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn [options] .. MFiX-Exa uses the same compiler flags used to build AMReX, unless ``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or diff --git a/docs/source/InputsMonitors.rst b/docs/source/InputsMonitors.rst index 3d54a4a1d20c83d2984c9b314323d715603c44d2..580cea30c22ff5b3804cdc70e4da91a6c26dcf1e 100644 --- a/docs/source/InputsMonitors.rst +++ b/docs/source/InputsMonitors.rst @@ -4,16 +4,16 @@ Monitors ======== The following inputs must be preceded by "amr" and control whether to compute -monitors, i.e., spatial averages, and how often to output the results. -n is the number of monitors implicitly defined by the size of avg_region_x_w. +monitors, i.e., spatial averages, and how often to output the results. +n is the number of monitors implicitly defined by the size of avg_region_x_w. +------------------+-----------------------------------------------------------------------+-------------+-----------+ | | Description | Type | Default | +==================+=======================================================================+=============+===========+ | avg_int | Interval, in number of CFD dt's, to write output | Int | -1 | +------------------+-----------------------------------------------------------------------+-------------+-----------+ -| avg_file | Base file name which is appended with the data type (vel_p_, p_g_, | String | avg_region| -| | ep_g_ or vel_g_), the number of this type of averaging, and the .csv | | | +| avg_file | Base file name which is appended with the data type (vel_p, p_g, | String | avg_region| +| | ep_g or vel_g), the number of this type of averaging, and the .csv | | | | | file extension | | | +------------------+-----------------------------------------------------------------------+-------------+-----------+ | avg_vel_p | Average and save particle velocity (if 1) | n*Int | 0 | @@ -36,6 +36,3 @@ n is the number of monitors implicitly defined by the size of avg_region_x_w. +------------------+-----------------------------------------------------------------------+-------------+-----------+ | avg_region_z_t | Upper bound of averaging region in z-direction | n*Real | None | +------------------+-----------------------------------------------------------------------+-------------+-----------+ - - - diff --git a/docs/source/InputsProblemDefinition.rst b/docs/source/InputsProblemDefinition.rst index bd6c14c7f0dca0719e8e66913405d0b933efce85..66fef77f7509f2028d3723a9da8ef34640e100d7 100644 --- a/docs/source/InputsProblemDefinition.rst +++ b/docs/source/InputsProblemDefinition.rst @@ -27,12 +27,12 @@ The following inputs must be preceded by "geometry." The following inputs must be preceded by "mfix." -+----------------------+-------------------------------------------------------------------------+-------------+--------+ ++----------------------+-------------------------------------------------------------------------+----------+-----------+ | | Description | Type | Default | +======================+=========================================================================+==========+===========+ | geometry | Which type of EB geometry are we using? | String | | +----------------------+-------------------------------------------------------------------------+----------+-----------+ -| levelset__refinement | Refinement factor of levelset resolution relative to level 0 resolution | Int | 1 ! +| levelset__refinement | Refinement factor of levelset resolution relative to level 0 resolution | Int | 1 | +----------------------+-------------------------------------------------------------------------+----------+-----------+ Setting basic EB walls can be specified by inputs preceded by "xlo", "xhi", "ylo", "yhi", "zlo", and "zhi" diff --git a/docs/source/LoadBalancing.rst b/docs/source/LoadBalancing.rst index 7496e3934c3b7ea16081b9e1f1e97057350dc60d..7c685d8617a693bcd8fc432e2389aebbfedbfe7c 100644 --- a/docs/source/LoadBalancing.rst +++ b/docs/source/LoadBalancing.rst @@ -9,23 +9,23 @@ Load Balancing -------------- -The process of load balancing is typically independent of the process of grid creation; -the inputs to load balancing are a given set of grids with a set of weights +The process of load balancing is typically independent of the process of grid creation; +the inputs to load balancing are a given set of grids with a set of weights assigned to each grid. (The exception to this is the KD-tree approach in which the grid creation process is governed by trying to balance the work in each grid.) -Single-level load balancing algorithms are sequentially applied to each AMR level independently, -and the resulting distributions are mapped onto the ranks taking into account the weights +Single-level load balancing algorithms are sequentially applied to each AMR level independently, +and the resulting distributions are mapped onto the ranks taking into account the weights already assigned to them (assign heaviest set of grids to the least loaded rank) Options supported by AMReX include: -- Knapsack: the default weight of a grid in the knapsack algorithm is the number of grid cells, - but AMReX supports the option to pass an array of weights – one per grid – or alternatively +- Knapsack: the default weight of a grid in the knapsack algorithm is the number of grid cells, + but AMReX supports the option to pass an array of weights – one per grid – or alternatively to pass in a MultiFab of weights per cell which is used to compute the weight per grid -- SFC: enumerate grids with a space-filling Z-morton curve, then partition the +- SFC: enumerate grids with a space-filling Z-morton curve, then partition the resulting ordering across ranks in a way that balances the load - Round-robin: sort grids and assign them to ranks in round-robin fashion -- specifically - FAB *i* is owned by CPU *i*%N where N is the total number of MPI ranks. + FAB ``i`` is owned by CPU ``i % N`` where N is the total number of MPI ranks.