Commit 535a57d0 authored by Ann Almgren's avatar Ann Almgren
Browse files

Merge branch 'master' of https://github.com/AMReX-Codes/MFIX-Exa

parents 0e4f28f1 7a17bb42
Loading
Loading
Loading
Loading
+105 −0
Original line number Diff line number Diff line
@@ -36,6 +36,7 @@ must meet MFiX-Exa requirements.
.. note::
   **MFiX-Exa requires CMake 3.14 or higher.**

.. _sec:build:superbuild:

SUPERBUILD Instructions (recommended)
-------------------------------------
@@ -220,3 +221,107 @@ the above command (with full path to the compilers) or the following:
MFiX-Exa uses the same compiler flags used to build AMReX, unless
``CMAKE_Fortran_FLAGS``/``CMAKE_CXX_FLAGS`` is explicitly provided, or
the environmental variables ``FFLAGS``/``CXXFLAGS`` are set.


Building MFiX-Exa for Cori (NERSC)
-----------------------------------

Standard build 
~~~~~~~~~~~~~~~~~~~

For the Cori cluster at NERSC, you first need to load/unload modules required to build MFIX-Exa.

.. code:: shell

    > module unload altd
    > module unload darshan
    > module load cmake/3.14.0

The default options for Cori are the **Haswell** architecture and **Intel** compiler, if you want to compile with the **Knight's Landing (KNL)** architecture:

.. code:: shell

    > module swap craype-haswell craype-mic-knl

Or use the **GNU** compiler:

.. code:: shell

    > module swap PrgEnv-intel PrgEnv-gnu

Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`.

.. note::

    The load/unload modules options could be saved in the `~/.bash_profile.ext` 


GPU build 
~~~~~~~~~~~~~~~~~~~

To compile on the GPU nodes in Cori, you first need to purge your modules, most of which won't work on the GPU nodes

.. code:: shell

    > module purge

Then, you need to load the following modules:

.. code:: shell

    > module load modules esslurm gcc cuda openmpi/3.1.0-ucx cmake/3.14.0

Currently, you need to use OpenMPI; mvapich2 seems not to work.

Then, you need to use slurm to request access to a GPU node:

.. code:: shell

    > salloc -N 1 -t 02:00:00 -c 80 -C gpu -A m1759 --gres=gpu:8 --exclusive

This reservers an entire GPU node for your job. Note that you can’t cross-compile for the GPU nodes - you have to log on to one and then build your software.

Finally, navigate to the base of the MFIX-Exa repository and compile in GPU mode:

.. code:: shell

    > cd mfix
    > mdkir build
    > cd build
    > cmake -DENABLE_CUDA=yes -DAMREX_CUDA_ARCH=Volta -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran ..
    > make -j

For more information about GPU nodes in Cori -- `<https://docs-dev.nersc.gov/cgpu/>`_

Building MFiX-Exa for Summit (OLCF)
-----------------------------------

For the Summit cluster at OLCF, you first need to load/unload modules required to build MFIX-Exa.

.. code:: shell

    > module unload xalt
    > module unload darshan
    > module load gcc
    > module load cmake/3.14.0

Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`.

To build MFIX-Exa for GPUs, you need to load cuda module:

.. code:: shell

    > module load cuda/10.1.105

To compile for GPUs:

.. code:: shell

    > cd mfix
    > mdkir build
    > cd build
    > cmake -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DENABLE_CUDA=yes 
    > make -j

An example of a *submission_script* for GPUs can be found in the repo ``mfix/tests/GPU_test/script.sh``.
For more information about Summit cluster: `<https://www.olcf.ornl.gov/for-users/system-user-guides/summit/>`_ 
+24 −16
Original line number Diff line number Diff line
@@ -20,16 +20,16 @@ These control the nodal projection and must be preceded by "mfix":
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_atol                 |  Absolute tolerance in nodal projection                               |    Real     |   1.e-14     | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_maxiter              |  Maximum number of iterations in the nodal projection                 |    Int      |              | 
| mg_maxiter              |  Maximum number of iterations in the nodal projection                 |    Int      |   100        | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_cg_maxiter           |  Maximum number of iterations in the nodal projection                 |    Int      |              | 
| mg_cg_maxiter           |  Maximum number of iterations in the nodal projection                 |    Int      |   100        | 
|                         |  bottom solver if using bicg, cg, bicgcg or cgbicg                    |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_max_coarsening_level |  Maximum number of coarser levels to allowin the nodal projection     |    Int      |              | 
| mg_max_coarsening_level |  Maximum number of coarser levels to allowin the nodal projection     |    Int      |   100        | 
|                         |  If set to 0, the bottom solver will be called at the current level   |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver_type      |  Which bottom solver to use in the nodal projection                   |  String     |   bicgcg     |
|                         |  Options are bicgstab, cg, cgbicg, smoother or hypre                  |             |              | 
|                         |  Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre          |             |              | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+

These control the MAC projection and must be preceded by "mac":
@@ -45,32 +45,40 @@ These control the MAC projection and must be preceded by "mac":
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_atol                 |  Absolute tolerance in MAC projection                                 |    Real     |   1.e-14     | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_maxiter              |  Maximum number of iterations in the MAC projection                   |    Int      |              | 
| mg_maxiter              |  Maximum number of iterations in the MAC projection                   |    Int      |   200        | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_cg_maxiter           |  Maximum number of iterations in the MAC projection                   |    Int      |              | 
| mg_cg_maxiter           |  Maximum number of iterations in the MAC projection                   |    Int      |   200        | 
|                         |  bottom solver if using bicg, cg, bicgcg or cgbicg                    |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_max_coarsening_level |  Maximum number of coarser levels to allow in the nodal projection    |    Int      |              | 
| mg_max_coarsening_level |  Maximum number of coarser levels to allow in the MAC projection      |    Int      |   100        | 
|                         |  If set to 0, the bottom solver will be called at the current level   |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver_type      |  Which bottom solver to use in the MAC projection                     |  String     |   bicgcg     |
|                         |  Options are bicgstab, cg, cgbicg, smoother or hypre                  |             |              | 
|                         |  Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre          |             |              | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+

These control the diffusion solver and must be preceded by "diff":
The following inputs must be preceded by "diff" 

+----------------------+-----------------------------------------------------------------------+-------------+--------------+
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
|                         | Description                                                           |   Type      | Default      |
+======================+=======================================================================+=============+==============+
+=========================+=======================================================================+=============+==============+
| mg_verbose              |  Verbosity of linear solver for diffusion solve                       |    Int      |   0          |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_cg_verbose           |  Verbosity of BiCGStab solver in diffusion solve                      |    Int      |   0          |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_rtol                 |  Relative tolerance in diffusion solve                                |    Real     |   1.e-11     | 
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_atol                 |  Absolute tolerance in diffusion solve                                |    Real     |   1.e-14     | 
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_maxiter              |  Maximum number of iterations in diffusion solve                      |    Int      |   100        |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_cg_maxiter           |  Maximum number of iterations in diffusion solve                      |    Int      |   100        |
|                         |  bottom solver if using bicg, cg, bicgcg or cgbicg                    |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| mg_max_coarsening_level |  Maximum number of coarser levels to allow in diffusion solve         |    Int      |   100        |
|                         |  If set to 0, the bottom solver will be called at the current level   |             |              |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver_type      |  Which bottom solver to use in the diffusion solve                    |  String     |   bicgcg     |
|                      |  Options are bicgstab, cg, cgbicg, smoother or hypre                  |             |              | 
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
|                         |  Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre          |             |              | 
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
+1 −1
Original line number Diff line number Diff line
@@ -43,7 +43,7 @@ Currently, the criterion for setting "steady_state" to true is if "dt" is undefi
+-----------------------+-----------------------------------------------------------------------+-------------+------------+
| steady_state_tol      | Tolerance for checking if we have reached steady state                |   Real      | None       |
|                       |                                                                       |             |            |
|                       | (Must be set if steady_state_tol = 1)                                 |             |            |
|                       | (Must be set if steady_state = 1)                                     |             |            |
+-----------------------+-----------------------------------------------------------------------+-------------+------------+
| steady_state_maxiter  | Maximum number of allowed iterations to converge to steady state      |   Int       | 100000000  |
+-----------------------+-----------------------------------------------------------------------+-------------+------------+