diff --git a/docs/source/BuildingCMake.rst b/docs/source/BuildingCMake.rst index a881da6e8486931ad40efb6d93d5d86d7d1101d2..5b8af335257f3571b76315e0423a909d77dfc94a 100644 --- a/docs/source/BuildingCMake.rst +++ b/docs/source/BuildingCMake.rst @@ -36,6 +36,7 @@ must meet MFiX-Exa requirements. .. note:: **MFiX-Exa requires CMake 3.14 or higher.** +.. _sec:build:superbuild: SUPERBUILD Instructions (recommended) ------------------------------------- @@ -220,3 +221,107 @@ the above command (with full path to the compilers) or the following: MFiX-Exa uses the same compiler flags used to build AMReX, unless ``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or the environmental variables ``FFLAGS``/``CXXFLAGS`` are set. + + +Building MFiX-Exa for Cori (NERSC) +----------------------------------- + +Standard build +~~~~~~~~~~~~~~~~~~~ + +For the Cori cluster at NERSC, you first need to load/unload modules required to build MFIX-Exa. + +.. code:: shell + + > module unload altd + > module unload darshan + > module load cmake/3.14.0 + +The default options for Cori are the **Haswell** architecture and **Intel** compiler, if you want to compile with the **Knight's Landing (KNL)** architecture: + +.. code:: shell + + > module swap craype-haswell craype-mic-knl + +Or use the **GNU** compiler: + +.. code:: shell + + > module swap PrgEnv-intel PrgEnv-gnu + +Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`. + +.. note:: + + The load/unload modules options could be saved in the `~/.bash_profile.ext` + + +GPU build +~~~~~~~~~~~~~~~~~~~ + +To compile on the GPU nodes in Cori, you first need to purge your modules, most of which won't work on the GPU nodes + +.. code:: shell + + > module purge + +Then, you need to load the following modules: + +.. code:: shell + + > module load modules esslurm gcc cuda openmpi/3.1.0-ucx cmake/3.14.0 + +Currently, you need to use OpenMPI; mvapich2 seems not to work. + +Then, you need to use slurm to request access to a GPU node: + +.. code:: shell + + > salloc -N 1 -t 02:00:00 -c 80 -C gpu -A m1759 --gres=gpu:8 --exclusive + +This reservers an entire GPU node for your job. Note that you can’t cross-compile for the GPU nodes - you have to log on to one and then build your software. + +Finally, navigate to the base of the MFIX-Exa repository and compile in GPU mode: + +.. code:: shell + + > cd mfix + > mdkir build + > cd build + > cmake -DENABLE_CUDA=yes -DAMREX_CUDA_ARCH=Volta -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran .. + > make -j + +For more information about GPU nodes in Cori -- ``_ + +Building MFiX-Exa for Summit (OLCF) +----------------------------------- + +For the Summit cluster at OLCF, you first need to load/unload modules required to build MFIX-Exa. + +.. code:: shell + + > module unload xalt + > module unload darshan + > module load gcc + > module load cmake/3.14.0 + +Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`. + +To build MFIX-Exa for GPUs, you need to load cuda module: + +.. code:: shell + + > module load cuda/10.1.105 + +To compile for GPUs: + +.. code:: shell + + > cd mfix + > mdkir build + > cd build + > cmake -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DENABLE_CUDA=yes + > make -j + +An example of a *submission_script* for GPUs can be found in the repo ``mfix/tests/GPU_test/script.sh``. +For more information about Summit cluster: ``_ diff --git a/docs/source/InputsMultigrid.rst b/docs/source/InputsMultigrid.rst index 7ed2a9fb4dbe85276443f783b0327d4c6cceb9bc..0a90b33fd43e0db45611927bc94e8542b57bd079 100644 --- a/docs/source/InputsMultigrid.rst +++ b/docs/source/InputsMultigrid.rst @@ -20,16 +20,16 @@ These control the nodal projection and must be preceded by "mfix": +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ | mg_atol | Absolute tolerance in nodal projection | Real | 1.e-14 | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_maxiter | Maximum number of iterations in the nodal projection | Int | | +| mg_maxiter | Maximum number of iterations in the nodal projection | Int | 100 | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_cg_maxiter | Maximum number of iterations in the nodal projection | Int | | +| mg_cg_maxiter | Maximum number of iterations in the nodal projection | Int | 100 | | | bottom solver if using bicg, cg, bicgcg or cgbicg | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_max_coarsening_level | Maximum number of coarser levels to allowin the nodal projection | Int | | +| mg_max_coarsening_level | Maximum number of coarser levels to allowin the nodal projection | Int | 100 | | | If set to 0, the bottom solver will be called at the current level | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ | bottom_solver_type | Which bottom solver to use in the nodal projection | String | bicgcg | -| | Options are bicgstab, cg, cgbicg, smoother or hypre | | | +| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ These control the MAC projection and must be preceded by "mac": @@ -45,32 +45,40 @@ These control the MAC projection and must be preceded by "mac": +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ | mg_atol | Absolute tolerance in MAC projection | Real | 1.e-14 | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_maxiter | Maximum number of iterations in the MAC projection | Int | | +| mg_maxiter | Maximum number of iterations in the MAC projection | Int | 200 | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_cg_maxiter | Maximum number of iterations in the MAC projection | Int | | +| mg_cg_maxiter | Maximum number of iterations in the MAC projection | Int | 200 | | | bottom solver if using bicg, cg, bicgcg or cgbicg | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_max_coarsening_level | Maximum number of coarser levels to allow in the nodal projection | Int | | +| mg_max_coarsening_level | Maximum number of coarser levels to allow in the MAC projection | Int | 100 | | | If set to 0, the bottom solver will be called at the current level | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ | bottom_solver_type | Which bottom solver to use in the MAC projection | String | bicgcg | -| | Options are bicgstab, cg, cgbicg, smoother or hypre | | | +| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | | +-------------------------+-----------------------------------------------------------------------+-------------+--------------+ These control the diffusion solver and must be preceded by "diff": The following inputs must be preceded by "diff" -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| | Description | Type | Default | -+======================+=======================================================================+=============+==============+ -| mg_verbose | Verbosity of linear solver for diffusion solve | Int | 0 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_cg_verbose | Verbosity of BiCGStab solver in diffusion solve | Int | 0 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_rtol | Relative tolerance in diffusion solve | Real | 1.e-11 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| mg_atol | Absolute tolerance in diffusion solve | Real | 1.e-14 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| bottom_solver_type | Which bottom solver to use in the diffusion solve | String | bicgcg | -| | Options are bicgstab, cg, cgbicg, smoother or hypre | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| | Description | Type | Default | ++=========================+=======================================================================+=============+==============+ +| mg_verbose | Verbosity of linear solver for diffusion solve | Int | 0 | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_cg_verbose | Verbosity of BiCGStab solver in diffusion solve | Int | 0 | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_rtol | Relative tolerance in diffusion solve | Real | 1.e-11 | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_atol | Absolute tolerance in diffusion solve | Real | 1.e-14 | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_maxiter | Maximum number of iterations in diffusion solve | Int | 100 | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_cg_maxiter | Maximum number of iterations in diffusion solve | Int | 100 | +| | bottom solver if using bicg, cg, bicgcg or cgbicg | | | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| mg_max_coarsening_level | Maximum number of coarser levels to allow in diffusion solve | Int | 100 | +| | If set to 0, the bottom solver will be called at the current level | | | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| bottom_solver_type | Which bottom solver to use in the diffusion solve | String | bicgcg | +| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | | ++-------------------------+-----------------------------------------------------------------------+-------------+--------------+ diff --git a/docs/source/InputsTimeStepping.rst b/docs/source/InputsTimeStepping.rst index 3c194b5358e0f42f66b38c9752e5735ea7b2d171..5d2be8e2e1685d6452c561e2232ab4f779469641 100644 --- a/docs/source/InputsTimeStepping.rst +++ b/docs/source/InputsTimeStepping.rst @@ -43,7 +43,7 @@ Currently, the criterion for setting "steady_state" to true is if "dt" is undefi +-----------------------+-----------------------------------------------------------------------+-------------+------------+ | steady_state_tol | Tolerance for checking if we have reached steady state | Real | None | | | | | | -| | (Must be set if steady_state_tol = 1) | | | +| | (Must be set if steady_state = 1) | | | +-----------------------+-----------------------------------------------------------------------+-------------+------------+ | steady_state_maxiter | Maximum number of allowed iterations to converge to steady state | Int | 100000000 | +-----------------------+-----------------------------------------------------------------------+-------------+------------+