diff --git a/docs/source_docs/references/hpc/Delta.rst b/docs/source_docs/references/hpc/Delta.rst index 6b7e5e69f7d62fbbaab85e8d76970ce34cf78678..7af2fd6f047f3bb255e37ce67dd84aeaea311660 100644 --- a/docs/source_docs/references/hpc/Delta.rst +++ b/docs/source_docs/references/hpc/Delta.rst @@ -1,502 +1,500 @@ -NCSA Delta -========== - -.. warning:: - - There is a known and, thus far, unresolved issue when building - with cmake for GPU and hypre support. - -If this is your first time building MFIX-Exa on Delta, please -review the general notes below and `Basics`_ section first. - -* To access Delta, you will need an NCSA account which can be administered - by the NSF ACCESS allocation management system. -* You can find the name of your account(s) and the current balance with - the ``accounts`` command. -* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a ``build`` directory. - The gmake instructions compile to a ``exec`` directory. -* For the dependencies, it is assumed that you have set the - following environment variables: - - .. code:: bash - - export HYPRE_INSTALL_DIR=$HOME/ - export CSG_INSTALL_DIR=$HOME/ - export CSG_LIB_DIR=$HOME/ - export ASCENT_INSTALL_DIR=$HOME/ - - to a path that you have read/write access to. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. -* After building the ``mfix`` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command - in the ``build`` directory - - .. code:: bash - - cmake --build . --target pic2dem - - -Basics ------- - -Source code -~~~~~~~~~~~ - -Before building, first obtain a copy of the source code -following the instructions on the -`MFIX-Exa website. `_ - -Modules -~~~~~~~ - -The default modules on Delta are *almost* all that you need to build -and compile all dependencies. If you are using ``cmake`` to build, you -need to load a newer version, such as - -.. code:: bash - - module load cmake/3.23.1 - -If you are building the CSG library or building MFIX-Exa with CSG support, -you need to load boost, such as - -.. code:: bash - - module load boost/1.80.0 - -There is a known incompatibility with AMReX and the pre-loaded cuda version 11.6.1. -To build with GPU support you will need to downgrade or upgrade versions, such as - -.. code:: bash - - module swap cuda/11.6.1 cuda/11.7.0 - -Full builds that utilize external dependencies, also require setting -certain environment variables as discussed below. - - -Building MFIX-Exa ------------------ - -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, -you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. There are two primary -methods of building the code ``cmake`` and ``gmake`` which are provided -separately below. - -cmake -~~~~~ - -.. tabs:: - - .. tab:: CPU - - .. code:: bash - - cmake -DMFIX_MPI=yes \ - -DMFIX_OMP=no \ - -DMFIX_GPU_BACKEND=NONE \ - -DAMReX_TINY_PROFILE=no \ - -DMFIX_CSG=no \ - -DMFIX_HYPRE=no \ - -DCMAKE_BUILD_TYPE=Release \ - ../ - make -j8 - - .. tab:: GPU - - .. code:: bash - - cmake -DMFIX_MPI=yes \ - -DMFIX_OMP=no \ - -DMFIX_CSG=no \ - -DMFIX_HYPRE=no \ - -DMFIX_GPU_BACKEND=CUDA \ - -DAMReX_CUDA_ARCH=8.0 \ - -DCMAKE_CUDA_ARCHITECTURES="80" \ - -DGPUS_PER_NODE=4 \ - -DAMReX_TINY_PROFILE=no \ - -DCMAKE_BUILD_TYPE=Release \ - ../ - make -j8 - - .. tab:: CPU-full - - .. code:: bash - - export HYPRE_DIR=$HYPRE_INSTALL_DIR - export HYPRE_ROOT=$HYPRE_DIR - export HYPRE_LIBRARIES=$HYPRE_DIR/lib - export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include - - export ASCENT_DIR=$ASCENT_INSTALL_DIR - export CONDUIT_DIR=$ASCENT_DIR - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit - - export CSG_DIR=$CSG_INSTALL_DIR - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - - cmake -DMFIX_MPI=yes \ - -DMFIX_OMP=no \ - -DMFIX_CSG=yes \ - -DMFIX_HYPRE=yes \ - -DAMReX_ASCENT=yes \ - -DAMReX_CONDUIT=yes \ - -DMFIX_GPU_BACKEND=NONE \ - -DAMReX_TINY_PROFILE=no \ - -DCMAKE_BUILD_TYPE=Release \ - ../ - make -j8 - - .. tab:: GPU-full - - .. code:: bash - - export HYPRE_DIR=$HYPRE_INSTALL_DIR - export HYPRE_ROOT=$HYPRE_DIR - export HYPRE_LIBRARIES=$HYPRE_DIR/lib - export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include - - export ASCENT_DIR=$ASCENT_INSTALL_DIR - export CONDUIT_DIR=$ASCENT_DIR - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit - - export CSG_DIR=$CSG_INSTALL_DIR - export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - - cmake -DBoost_INCLUDE_DIR="$BOOST_ROOT/include" \ - -DMFIX_MPI=yes \ - -DMFIX_OMP=no \ - -DMFIX_CSG=yes \ - -DMFIX_HYPRE=yes \ - -DAMReX_ASCENT=yes \ - -DAMReX_CONDUIT=yes \ - -DMFIX_GPU_BACKEND=CUDA \ - -DAMReX_CUDA_ARCH=8.0 \ - -DCMAKE_CUDA_ARCHITECTURES="80" \ - -DGPUS_PER_SOCKET=1 \ - -DGPUS_PER_NODE=2 \ - -DAMReX_TINY_PROFILE=no \ - -DCMAKE_BUILD_TYPE=Release \ - ../ - make -j8 - - -gmake -~~~~~ - -.. tabs:: - - .. tab:: CPU - - .. code:: bash - - make -C exec -j8 \ - COMP=gnu \ - USE_MPI=TRUE \ - USE_OMP=FALSE \ - USE_CUDA=FALSE \ - USE_TINY_PROFILE=FALSE \ - USE_CSG=FALSE \ - USE_HYPRE=FALSE \ - DEBUG=FALSE - - - .. tab:: GPU - - .. code:: bash - - make -C exec -j8 - COMP=gnu \ - USE_MPI=TRUE \ - USE_OMP=FALSE \ - USE_CUDA=TRUE \ - CUDA_ARCH=8.0 \ - USE_TINY_PROFILE=FALSE \ - USE_CSG=FALSE \ - USE_HYPRE=FALSE \ - DEBUG=FALSE - - - .. tab:: CPU-full - - .. code:: bash - - export HYPRE_DIR=$HYPRE_INSTALL_DIR - export HYPRE_HOME=$HYPRE_DIR - - export ASCENT_DIR=$ASCENT_INSTALL_DIR - export CONDUIT_DIR=$ASCENT_DIR - - export CSGEB_HOME=$CSG_LIB_DIR - export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib" - - make -C exec -j8 \ - COMP=gnu \ - USE_MPI=TRUE \ - USE_OMP=FALSE \ - USE_CUDA=FALSE \ - USE_TINY_PROFILE=FALSE \ - USE_CSG=TRUE \ - USE_HYPRE=TRUE \ - USE_ASCENT=TRUE \ - USE_CONDUIT=TRUE \ - DEBUG=FALSE - - - .. tab:: GPU-full - - .. code:: bash - - export HYPRE_DIR=$HYPRE_INSTALL_DIR - export HYPRE_HOME=$HYPRE_DIR - - export ASCENT_DIR=$ASCENT_INSTALL_DIR - export CONDUIT_DIR=$ASCENT_DIR - - export CSGEB_HOME=$CSG_LIB_DIR - export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib" - - make -C exec -j8 COMP=gnu \ - USE_MPI=TRUE \ - USE_OMP=FALSE \ - USE_CUDA=TRUE \ - CUDA_ARCH=8.0 \ - USE_TINY_PROFILE=FALSE \ - USE_CSG=TRUE \ - USE_HYPRE=TRUE \ - USE_ASCENT=TRUE \ - USE_CONDUIT=TRUE \ - DEBUG=FALSE - - - - -Optional build dependencies ---------------------------- - -The following dependencies need to be built and installed -prior to following any of the full build instructions above. - -#. HYPRE - - .. tabs:: - - .. tab:: CPU - - .. code:: bash - - git clone https://github.com/hypre-space/hypre.git - pushd hypre/src/ - git checkout v2.26.0 - ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install - popd - - .. tab:: GPU - - .. code:: bash - - git clone https://github.com/hypre-space/hypre.git - pushd hypre/src/ - git checkout v2.26.0 - ./configure --prefix=$HYPRE_INSTALL_DIR \ - --without-superlu \ - --disable-bigint \ - --without-openmp \ - --with-MPI \ - --with-cuda \ - --with-gpu-arch='80' \ - --with-cuda-home=$CUDA_HOME \ - --enable-cusparse \ - --enable-curand - make -j8 install - popd - -#. Catch2 - - .. code:: bash - - git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2 - pushd Catch2/ - cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR - cd build/ - make -j8 install - popd - -#. GMP - - .. code:: bash - - wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz - tar -xf gmp-6.2.1.tar.xz - pushd gmp-6.2.1 - ./configure --prefix=$CSG_INSTALL_DIR - make -j8 install - popd - -#. MPFR - - .. code:: bash - - wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz - tar -xf mpfr-4.1.0.tar.xz - pushd mpfr-4.1.0/ - ./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR - make -j8 install - popd - -#. CGAL - - .. code:: bash - - git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal - pushd cgal/ - cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR - cd build/ - make -j8 install - popd - - -#. PEGTL - - .. code:: bash - - git clone --branch 3.2.2 https://github.com/taocpp/PEGTL - pushd PEGTL/ - cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR - cd build/ - make -j8 install - popd - -#. CSG EB library (**gmake**) - - When building MFIX-Exa with CSG support using the ``gmake`` build option, - you need to install ``libcsgeb`` to ``$CSG_LIB_DIR``. This can be done - using ``cmake`` or ``gmake``, but since you are using ``gmake`` to build - MFIX-Exa, let's just assume you will opt for ``gmake`` here as well. - Make sure you have boost loaded before building. (This is intended to be - executed from inside the mfix repo.) - - .. code:: bash - - make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \ - PEGTL_HOME=$CSG_INSTALL_DIR \ - CGAL_HOME=$CSG_INSTALL_DIR \ - CATCH2_HOME=$CSG_INSTALL_DIR \ - ENABLE_CGAL=TRUE - -#. Conduit - - .. code:: bash - - git clone --recursive https://github.com/LLNL/conduit.git - pushd conduit/ - git checkout v0.8.6 - mkdir build && cd build - cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ - -DENABLE_OPENMP=OFF \ - -DENABLE_MPI=ON \ - -DENABLE_CUDA=OFF \ - -DCMAKE_BUILD_TYPE=Release - make -j8 install - popd - -#. Vtk-m - - .. code:: bash - - git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git - pushd vtk-m/ - git checkout v1.9.0 - mkdir build && cd build/ - cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ - -DVTKm_ENABLE_OPENMP=OFF \ - -DVTKm_ENABLE_MPI=ON \ - -DVTKm_ENABLE_CUDA=OFF \ - -DVTKm_USE_64BIT_IDS=OFF \ - -DVTKm_USE_DOUBLE_PRECISION=ON \ - -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \ - -DVTKm_NO_DEPRECATED_VIRTUAL=ON \ - -DCMAKE_BUILD_TYPE=Release - make -j8 install - popd - -#. Ascent - - .. code:: bash - - git clone --recursive https://github.com/Alpine-DAV/ascent.git - pushd ascent - git checkout v0.9.0 - mkdir build && cd build/ - cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ - -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \ - -DVTKM_DIR=$ASCENT_INSTALL_DIR \ - -DENABLE_VTKH=ON \ - -DENABLE_FORTRAN=OFF \ - -DENABLE_PYTHON=OFF \ - -DENABLE_DOCS=OFF \ - -DBUILD_SHARED_LIBS=ON \ - -DCMAKE_BUILD_TYPE=Release \ - -DENABLE_GTEST=OFF \ - -DENABLE_TESTS=OFF - make -j8 install - popd - - -Running Jobs ------------- - -Common Slurm commands: - -* ``sinfo`` see available/allocated resources -* ``sbatch runit_cpu.sh`` submit a cpu job to the queue -* ``squeue -u USER`` check job status of user USER -* ``squeue -p PARTITION`` check job status of partition PARTITION -* ``scancel JOBID`` kill a job with id JOBID - ``salloc -N 1 -p gpuA100x4 -A bbsj-delta-gpu --time=00:20:00 --exclusive --gpus-per-node=4`` grab a whole GPU node interactively for 20 minutes - -Example run script for GPU is below, -CPU-only runs have not been tested on this machine. - -.. code:: bash - - #!/bin/bash - #SBATCH --nodes=3 - #SBATCH --exclusive - #SBATCH --ntasks-per-node=4 - #SBATCH --cpus-per-task=16 # <- match to OMP_NUM_THREADS - #SBATCH --partition=gpuA100x4 # <- or one of: gpuA100x4 gpuA40x4 gpuA100x8 gpuMI100x8 - #SBATCH --account=bbsj-delta-gpu - #SBATCH --job-name=mympi - #SBATCH --time=00:05:00 # hh:mm:ss for the job - #SBATCH --constraint="scratch" - - ### GPU options ### - #SBATCH --gpus-per-node=4 - #SBATCH --gpus-per-task=1 - #SBATCH --gpu-bind=closest - #SBATCH --mail-user=first.last@institution.edu - #SBATCH --mail-type="BEGIN,END" - - module reset - module swap cuda/11.6.1 cuda/11.7.0 - module load boost/1.80.0 - module list - - echo "job is starting on `hostname`" - - srun -N 3 -n 12 -G 12 ./mfix inputs.rt > screen.txt - - +NCSA Delta +========== + +.. warning:: + + There is a known and, thus far, unresolved issue when building + with cmake for GPU and hypre support. + +If this is your first time building MFIX-Exa on Delta, please +review the general notes below and `Basics`_ section first. + +* To access Delta, you will need an NCSA account which can be administered + by the NSF ACCESS allocation management system. +* You can find the name of your account(s) and the current balance with + the ``accounts`` command. +* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a ``build`` directory. + The gmake instructions compile to a ``exec`` directory. +* For the dependencies, it is assumed that you have set the + following environment variables: + + .. code:: bash + + export HYPRE_INSTALL_DIR=$HOME/ + export CSG_INSTALL_DIR=$HOME/ + export CSG_LIB_DIR=$HOME/ + export ASCENT_INSTALL_DIR=$HOME/ + + to a path that you have read/write access to. + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. +* After building the ``mfix`` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command + in the ``build`` directory + + .. code:: bash + + cmake --build . --target pic2dem + + +Basics +------ + +Source code +~~~~~~~~~~~ + +Before building, first obtain a copy of the source code +following the instructions on the +`MFIX-Exa website. `_ + +Modules +~~~~~~~ + +The default modules on Delta are *almost* all that you need to build +and compile all dependencies. If you are using ``cmake`` to build, you +need to load a newer version, such as + +.. code:: bash + + module load cmake/3.23.1 + +If you are building the CSG library or building MFIX-Exa with CSG support, +you need to load boost, such as + +.. code:: bash + + module load boost/1.80.0 + +There is a known incompatibility with AMReX and the pre-loaded cuda version 11.6.1. +To build with GPU support you will need to downgrade or upgrade versions, such as + +.. code:: bash + + module swap cuda/11.6.1 cuda/11.7.0 + +Full builds that utilize external dependencies, also require setting +certain environment variables as discussed below. + + +Building MFIX-Exa +----------------- + +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, +you first need to build and install these libraries and their dependencies. +Instructions on building the necessary dependencies are below +and should be successfully installed first. There are two primary +methods of building MFIX-Exa: ``cmake`` and ``gmake`` which are provided +separately below. + +cmake +~~~~~ + +.. tabs:: + + .. tab:: CPU + + .. code:: bash + + cmake -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_GPU_BACKEND=NONE \ + -DAMReX_TINY_PROFILE=no \ + -DMFIX_CSG=no \ + -DMFIX_HYPRE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../ + make -j8 + + .. tab:: GPU + + .. code:: bash + + cmake -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_CSG=no \ + -DMFIX_HYPRE=no \ + -DMFIX_GPU_BACKEND=CUDA \ + -DAMReX_CUDA_ARCH=8.0 \ + -DCMAKE_CUDA_ARCHITECTURES="80" \ + -DGPUS_PER_NODE=4 \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../ + make -j8 + + .. tab:: CPU-full + + .. code:: bash + + export HYPRE_DIR=$HYPRE_INSTALL_DIR + export HYPRE_ROOT=$HYPRE_DIR + export HYPRE_LIBRARIES=$HYPRE_DIR/lib + export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include + + export ASCENT_DIR=$ASCENT_INSTALL_DIR + export CONDUIT_DIR=$ASCENT_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit + + export CSG_DIR=$CSG_INSTALL_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR + + cmake -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_CSG=yes \ + -DMFIX_HYPRE=yes \ + -DAMReX_ASCENT=yes \ + -DAMReX_CONDUIT=yes \ + -DMFIX_GPU_BACKEND=NONE \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../ + make -j8 + + .. tab:: GPU-full + + .. code:: bash + + export HYPRE_DIR=$HYPRE_INSTALL_DIR + export HYPRE_ROOT=$HYPRE_DIR + export HYPRE_LIBRARIES=$HYPRE_DIR/lib + export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include + + export ASCENT_DIR=$ASCENT_INSTALL_DIR + export CONDUIT_DIR=$ASCENT_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit + + export CSG_DIR=$CSG_INSTALL_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR + + cmake -DBoost_INCLUDE_DIR="$BOOST_ROOT/include" \ + -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_CSG=yes \ + -DMFIX_HYPRE=yes \ + -DAMReX_ASCENT=yes \ + -DAMReX_CONDUIT=yes \ + -DMFIX_GPU_BACKEND=CUDA \ + -DAMReX_CUDA_ARCH=8.0 \ + -DCMAKE_CUDA_ARCHITECTURES="80" \ + -DGPUS_PER_SOCKET=1 \ + -DGPUS_PER_NODE=2 \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../ + make -j8 + + +gmake +~~~~~ + +.. tabs:: + + .. tab:: CPU + + .. code:: bash + + make -C exec -j8 \ + COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=FALSE \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=FALSE \ + USE_HYPRE=FALSE \ + DEBUG=FALSE + + + .. tab:: GPU + + .. code:: bash + + make -C exec -j8 + COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=TRUE \ + CUDA_ARCH=8.0 \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=FALSE \ + USE_HYPRE=FALSE \ + DEBUG=FALSE + + + .. tab:: CPU-full + + .. code:: bash + + export HYPRE_DIR=$HYPRE_INSTALL_DIR + export HYPRE_HOME=$HYPRE_DIR + + export ASCENT_DIR=$ASCENT_INSTALL_DIR + export CONDUIT_DIR=$ASCENT_DIR + + export CSGEB_HOME=$CSG_LIB_DIR + export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib" + + make -C exec -j8 \ + COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=FALSE \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=TRUE \ + USE_HYPRE=TRUE \ + USE_ASCENT=TRUE \ + USE_CONDUIT=TRUE \ + DEBUG=FALSE + + + .. tab:: GPU-full + + .. code:: bash + + export HYPRE_DIR=$HYPRE_INSTALL_DIR + export HYPRE_HOME=$HYPRE_DIR + + export ASCENT_DIR=$ASCENT_INSTALL_DIR + export CONDUIT_DIR=$ASCENT_DIR + + export CSGEB_HOME=$CSG_LIB_DIR + export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib" + + make -C exec -j8 COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=TRUE \ + CUDA_ARCH=8.0 \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=TRUE \ + USE_HYPRE=TRUE \ + USE_ASCENT=TRUE \ + USE_CONDUIT=TRUE \ + DEBUG=FALSE + + + + +Optional build dependencies +--------------------------- + +The following dependencies need to be built and installed +prior to following any of the full build instructions above. + +#. HYPRE + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + git clone https://github.com/hypre-space/hypre.git + pushd hypre/src/ + git checkout v2.26.0 + ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI + make -j8 install + popd + + .. tab:: GPU + + .. code:: bash + + git clone https://github.com/hypre-space/hypre.git + pushd hypre/src/ + git checkout v2.26.0 + ./configure --prefix=$HYPRE_INSTALL_DIR \ + --without-superlu \ + --disable-bigint \ + --without-openmp \ + --with-MPI \ + --with-cuda \ + --with-gpu-arch='80' \ + --with-cuda-home=$CUDA_HOME \ + --enable-cusparse \ + --enable-curand + make -j8 install + popd + +#. Catch2 + + .. code:: bash + + git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2 + pushd Catch2/ + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR + cd build/ + make -j8 install + popd + +#. GMP + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz + tar -xf gmp-6.2.1.tar.xz + pushd gmp-6.2.1 + ./configure --prefix=$CSG_INSTALL_DIR + make -j8 install + popd + +#. MPFR + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz + tar -xf mpfr-4.1.0.tar.xz + pushd mpfr-4.1.0/ + ./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR + make -j8 install + popd + +#. CGAL + + .. code:: bash + + git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal + pushd cgal/ + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR + cd build/ + make -j8 install + popd + + +#. PEGTL + + .. code:: bash + + git clone --branch 3.2.2 https://github.com/taocpp/PEGTL + pushd PEGTL/ + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR + cd build/ + make -j8 install + popd + +#. CSG EB library (**gmake**) + + When building MFIX-Exa with CSG support using the ``gmake`` build option, + you need to install ``libcsgeb`` to ``$CSG_LIB_DIR``. This can be done + using ``cmake`` or ``gmake``, but since you are using ``gmake`` to build + MFIX-Exa, let's just assume you will opt for ``gmake`` here as well. + Make sure you have boost loaded before building. (This is intended to be + executed from inside the mfix repo.) + + .. code:: bash + + make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \ + PEGTL_HOME=$CSG_INSTALL_DIR \ + CGAL_HOME=$CSG_INSTALL_DIR \ + CATCH2_HOME=$CSG_INSTALL_DIR \ + ENABLE_CGAL=TRUE + +#. Conduit + + .. code:: bash + + git clone --recursive https://github.com/LLNL/conduit.git + pushd conduit/ + git checkout v0.8.6 + mkdir build && cd build + cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ + -DENABLE_OPENMP=OFF \ + -DENABLE_MPI=ON \ + -DENABLE_CUDA=OFF \ + -DCMAKE_BUILD_TYPE=Release + make -j8 install + popd + +#. Vtk-m + + .. code:: bash + + git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git + pushd vtk-m/ + git checkout v1.9.0 + mkdir build && cd build/ + cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ + -DVTKm_ENABLE_OPENMP=OFF \ + -DVTKm_ENABLE_MPI=ON \ + -DVTKm_ENABLE_CUDA=OFF \ + -DVTKm_USE_64BIT_IDS=OFF \ + -DVTKm_USE_DOUBLE_PRECISION=ON \ + -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \ + -DVTKm_NO_DEPRECATED_VIRTUAL=ON \ + -DCMAKE_BUILD_TYPE=Release + make -j8 install + popd + +#. Ascent + + .. code:: bash + + git clone --recursive https://github.com/Alpine-DAV/ascent.git + pushd ascent + git checkout v0.9.0 + mkdir build && cd build/ + cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ + -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \ + -DVTKM_DIR=$ASCENT_INSTALL_DIR \ + -DENABLE_VTKH=ON \ + -DENABLE_FORTRAN=OFF \ + -DENABLE_PYTHON=OFF \ + -DENABLE_DOCS=OFF \ + -DBUILD_SHARED_LIBS=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DENABLE_GTEST=OFF \ + -DENABLE_TESTS=OFF + make -j8 install + popd + + +Running Jobs +------------ + +Common Slurm commands: + +* ``sinfo`` see available/allocated resources +* ``sbatch runit_cpu.sh`` submit a cpu job to the queue +* ``squeue -u USER`` check job status of user USER +* ``squeue -p PARTITION`` check job status of partition PARTITION +* ``scancel JOBID`` kill a job with id JOBID + ``salloc -N 1 -p gpuA100x4 -A bbsj-delta-gpu --time=00:20:00 --exclusive --gpus-per-node=4`` grab a whole GPU node interactively for 20 minutes + +Example run script for GPU is below, +CPU-only runs have not been tested on this machine. + +.. code:: bash + + #!/bin/bash + #SBATCH --nodes=3 + #SBATCH --exclusive + #SBATCH --ntasks-per-node=4 + #SBATCH --cpus-per-task=16 # <- match to OMP_NUM_THREADS + #SBATCH --partition=gpuA100x4 # <- or one of: gpuA100x4 gpuA40x4 gpuA100x8 gpuMI100x8 + #SBATCH --account=bbsj-delta-gpu + #SBATCH --job-name=mympi + #SBATCH --time=00:05:00 # hh:mm:ss for the job + #SBATCH --constraint="scratch" + + ### GPU options ### + #SBATCH --gpus-per-node=4 + #SBATCH --gpus-per-task=1 + #SBATCH --gpu-bind=closest + #SBATCH --mail-user=first.last@institution.edu + #SBATCH --mail-type="BEGIN,END" + + module reset + module swap cuda/11.6.1 cuda/11.7.0 + module load boost/1.80.0 + module list + + echo "job is starting on `hostname`" + + srun -N 3 -n 12 -G 12 ./mfix inputs.rt > screen.txt diff --git a/docs/source_docs/references/hpc/Joule2.rst b/docs/source_docs/references/hpc/Joule2.rst index bf5232cf5dcffa47c463e09c1cf8d61a7ba5f4cc..1f0dd0618288abe5a832f105b4a76d8095a7edd7 100644 --- a/docs/source_docs/references/hpc/Joule2.rst +++ b/docs/source_docs/references/hpc/Joule2.rst @@ -1,19 +1,19 @@ Joule 2.0 ========== -If this is your first time building MFIX-Exa on Joule2, please +If this is your first time building MFIX-Exa on Joule2, please review the general notes below and `Basics`_ section first. -* To access Joule2, you will need an NETL HPC account, - a yubikey (2FA) and the +* To access Joule2, you will need an NETL HPC account, + a yubikey (2FA) and the `HPC client. `_ -* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a ``build`` directory. - The gmake instructions compile to a ``exec`` directory. -* For the dependencies, it is assumed that you have set the +* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a ``build`` directory. + The gmake instructions compile to a ``exec`` directory. +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -23,11 +23,11 @@ review the general notes below and `Basics`_ section first. export CSG_LIB_DIR=$HOME/ export ASCENT_INSTALL_DIR=$HOME/ - to a path that you have read/write access to. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. -* After building the ``mfix`` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command + to a path that you have read/write access to. + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. +* After building the ``mfix`` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the ``build`` directory .. code:: bash @@ -40,18 +40,18 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -All of the build instructions below have been tested with the +All of the build instructions below have been tested with the following modules -.. code:: bash +.. code:: bash module purge module load cmake/3.23.1 @@ -60,31 +60,31 @@ following modules The GPU-enabled builds additionally require -.. code:: bash +.. code:: bash module load cuda/11.3 -Full builds that utilize external dependencies, also require setting -certain environment variables as discussed below. +Full builds that utilize external dependencies, also require setting +certain environment variables as discussed below. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. There are two primary -methods of building the code ``cmake`` and ``gmake`` which are provided +Instructions on building the necessary dependencies are below +and should be successfully installed first. There are two primary +methods of building MFIX-Exa: ``cmake`` and ``gmake`` which are provided separately below. cmake ~~~~~ .. tabs:: - + .. tab:: CPU .. code:: bash @@ -197,9 +197,9 @@ cmake gmake ~~~~~ - + .. tabs:: - + .. tab:: CPU .. code:: bash @@ -213,13 +213,13 @@ gmake USE_CSG=FALSE \ USE_HYPRE=FALSE \ DEBUG=FALSE - + .. tab:: GPU .. code:: bash - - make -C exec -j8 + + make -C exec -j8 COMP=gnu \ USE_MPI=TRUE \ USE_OMP=FALSE \ @@ -260,7 +260,7 @@ gmake .. tab:: GPU-full .. code:: bash - + export HYPRE_DIR=$HYPRE_INSTALL_DIR export HYPRE_HOME=$HYPRE_DIR @@ -288,8 +288,8 @@ gmake Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. Set environment helpers @@ -300,7 +300,7 @@ prior to following any of the full build instructions above. export F77=$(which mpif77) export FC=$(which mpifort) export F90=$(which mpif90) - mkdir $HOME/scratch && cd $HOME/scratch + mkdir $HOME/scratch && cd $HOME/scratch #. HYPRE @@ -314,7 +314,7 @@ prior to following any of the full build instructions above. pushd hypre/src/ git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install + make -j8 install popd .. tab:: GPU @@ -335,7 +335,7 @@ prior to following any of the full build instructions above. --with-cuda-home=$CUDA_HOME \ --enable-cusparse \ --enable-curand - make -j8 install + make -j8 install popd #. Catch2 @@ -394,7 +394,7 @@ prior to following any of the full build instructions above. make -j8 install popd -#. CSG EB library (**gmake**) +#. CSG EB library (**gmake**) For the gmake install instructions, you need to install ``libcsgeb`` to ``$CSG_LIB_DIR`` using either cmake or gmake: @@ -499,13 +499,13 @@ Common Slurm commands: * ``squeue -u USER`` check job status of user USER * ``squeue -p PARTITION`` check job status of partition PARTITION * ``scancel JOBID`` kill a job with id JOBID -* ``salloc -N 1 -p gpu`` grab a GPU node interactively (for up to 48 hrs) +* ``salloc -N 1 -p gpu`` grab a GPU node interactively (for up to 48 hrs) * ``salloc -N 2 -p dev -q dev`` grab two development nodes (for up to 2 hrs) -Example run scripts: +Example run scripts: .. tabs:: - + .. tab:: CPU .. code:: bash @@ -515,19 +515,19 @@ Example run scripts: ##Accounting #SBATCH --partition=general #bigmem, dev #SBATCH --qos=normal #long, dev - + ##Submission #SBATCH --nodes=1 - #SBATCH --job-name="mfix-exa-run" + #SBATCH --job-name="mfix-exa-run" #SBATCH --output=job.out #SBATCH --mail-user=first.last@netl.doe.gov #SBATCH --mail-type=ALL - + ##Load Modules module purge module load gnu/9.3.0 module load openmpi/4.0.4_gnu9.3 - + ##Run the program mpirun -np 36 ./mfix inputs > screen.txt @@ -541,7 +541,7 @@ Example run scripts: ##Accounting #SBATCH --partition=gpu #SBATCH --qos=normal #long - + ##Submission #SBATCH --nodes=2 #SBATCH --ntasks-per-node=2 @@ -550,13 +550,12 @@ Example run scripts: #SBATCH --output=job.out #SBATCH --mail-user=first.last@netl.doe.gov #SBATCH --mail-type=ALL - + ##Load Modules module purge module load cuda/11.3 module load gnu/9.3.0 module load openmpi/4.0.4_gnu9.3 - + ##Run the program mpirun -np 4 -npersocket 1 ./mfix inputs > screen.txt - diff --git a/docs/source_docs/references/hpc/Perlmutter.rst b/docs/source_docs/references/hpc/Perlmutter.rst index 2f615d96b6ec2342bc0881fd809c76badbdc773f..674a0b95c304608ee95e9d4c6de322ba18128813 100644 --- a/docs/source_docs/references/hpc/Perlmutter.rst +++ b/docs/source_docs/references/hpc/Perlmutter.rst @@ -1,7 +1,7 @@ Perlmutter =========== -If this is your first time building MFIX-Exa on Perlmutter, please +If this is your first time building MFIX-Exa on Perlmutter, please review the general notes below and `Basics`_ section first. * The documentation for this system can be found `here. `_ @@ -14,13 +14,13 @@ review the general notes below and `Basics`_ section first. Login with your Iris password + (Google) Authenticator passcode. -* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a `build` directory. - The gmake instructions compile to a `exec` directory. -* For the dependencies, it is assumed that you have set the +* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a `build` directory. + The gmake instructions compile to a `exec` directory. +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -30,12 +30,12 @@ review the general notes below and `Basics`_ section first. export CSG_LIB_DIR=$HOME/ export ASCENT_INSTALL_DIR=$HOME/ - to a path that you have read/write access to, + to a path that you have read/write access to, such as inside the `$SCRATCH` space corresponding to your account. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. -* After building the `mfix` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. +* After building the `mfix` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the `build` directory .. code:: bash @@ -43,7 +43,7 @@ review the general notes below and `Basics`_ section first. cmake --build . --target pic2dem .. warning:: - + Currently, the ``cmake GPU build`` with hypre support links but does not run. We are still investigating, but recommend using ``gmake`` on Perlmutter at this time. @@ -52,18 +52,18 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -All of the build instructions below have been tested with the +All of the build instructions below have been tested with the following modules and environment helpers -.. code:: bash +.. code:: bash module load PrgEnv-gnu/8.3.3 module load cmake/3.22.0 @@ -74,7 +74,7 @@ following modules and environment helpers The GPU-enabled builds additionally require -.. code:: bash +.. code:: bash module load cudatoolkit/11.5 @@ -84,27 +84,27 @@ The GPU-enabled builds additionally require export MPICH_GPU_SUPPORT_ENABLED=1 export CRAY_ACCEL_TARGET=nvidia80 -Full builds that utilize external dependencies, also require setting -certain environment variables as discussed below. +Full builds that utilize external dependencies, also require setting +certain environment variables as discussed below. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. There are two primary -methods of building the code `cmake` and `gmake` which are provided +Instructions on building the necessary dependencies are below +and should be successfully installed first. There are two primary +methods of building MFIX-Exa: `cmake` and `gmake` which are provided separately below. cmake ~~~~~ .. tabs:: - + .. tab:: CPU .. code:: bash @@ -198,9 +198,9 @@ cmake gmake ~~~~~ - + .. tabs:: - + .. tab:: CPU .. code:: bash @@ -214,13 +214,13 @@ gmake USE_CSG=FALSE \ USE_HYPRE=FALSE \ DEBUG=FALSE - + .. tab:: GPU .. code:: bash - - make -C exec -j8 + + make -C exec -j8 COMP=gnu \ USE_MPI=TRUE \ USE_OMP=FALSE \ @@ -261,7 +261,7 @@ gmake .. tab:: GPU-full .. code:: bash - + export HYPRE_DIR=$HYPRE_INSTALL_DIR export HYPRE_HOME=$HYPRE_DIR @@ -289,8 +289,8 @@ gmake Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. HYPRE @@ -304,7 +304,7 @@ prior to following any of the full build instructions above. pushd hypre/src/ git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install + make -j8 install popd .. tab:: GPU @@ -325,7 +325,7 @@ prior to following any of the full build instructions above. --with-cuda-home=$CUDA_HOME \ --enable-cusparse \ --enable-curand - make -j8 install + make -j8 install popd #. Catch2 @@ -396,7 +396,7 @@ prior to following any of the full build instructions above. make -j8 install popd -#. CSG EB library (**gmake**) +#. CSG EB library (**gmake**) For the gmake install instructions, you need to install `libcsgeb` to `$CSG_LIB_DIR` using either cmake or gmake: @@ -498,10 +498,10 @@ Common Slurm commands: * **scancel JOBID** kill a job with id JOBID * **salloc --nodes 1 --qos interactive --time 01:00:00 --constraint gpu --gpus 4 --account=** grab 1 GPU node (for up to 1 hrs) -Example run scripts: +Example run scripts: .. tabs:: - + .. tab:: CPU .. code:: bash @@ -517,7 +517,7 @@ Example run scripts: #SBATCH --nodes=2 #SBATCH --ntasks-per-node=16 - # load modules used for build + # load modules used for build module load PrgEnv-gnu/8.3.3 module load cmake/3.22.0 @@ -543,7 +543,7 @@ Example run scripts: nrs=8 - # load modules used for build + # load modules used for build module load PrgEnv-gnu/8.3.3 module load cudatoolkit/11.5 module load cmake/3.22.0 diff --git a/docs/source_docs/references/hpc/Polaris.rst b/docs/source_docs/references/hpc/Polaris.rst index 7737701d37155ca83229fc5a6278b4046aebc539..5d07a63e77e2d420e4523856a1e4dd7b9bfd1891 100644 --- a/docs/source_docs/references/hpc/Polaris.rst +++ b/docs/source_docs/references/hpc/Polaris.rst @@ -1,7 +1,7 @@ Polaris ======== -If this is your first time building MFIX-Exa on Polaris, please +If this is your first time building MFIX-Exa on Polaris, please review the general notes below and `Basics`_ section first. * The documentation for this system can be found `here. `_ @@ -13,13 +13,13 @@ review the general notes below and `Basics`_ section first. Login with your Mobilepass token. No need to prefix or append with anything. -* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a `build` directory. - The gmake instructions compile to a `exec` directory. -* For the dependencies, it is assumed that you have set the +* These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a `build` directory. + The gmake instructions compile to a `exec` directory. +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -29,12 +29,12 @@ review the general notes below and `Basics`_ section first. export CSG_LIB_DIR=$HOME/ export ASCENT_INSTALL_DIR=$HOME/ - to a path that you have read/write access to, + to a path that you have read/write access to, such as inside the ``/lus/grand/projects//`` space corresponding to your account. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. -* After building the `mfix` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. +* After building the `mfix` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the `build` directory .. code:: bash @@ -46,18 +46,18 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -All of the build instructions below have been tested with the +All of the build instructions below have been tested with the following modules and environment helpers -.. code:: bash +.. code:: bash module swap PrgEnv-nvhpc PrgEnv-gnu module load cmake/3.23.2 @@ -65,33 +65,33 @@ following modules and environment helpers The GPU-enabled builds additionally require -.. code:: bash +.. code:: bash module load nvhpc-mixed export MPICH_GPU_SUPPORT_ENABLED=1 export AMREX_CUDA_ARCH=8.0 -Full builds that utilize external dependencies, also require setting -certain environment variables as discussed below. +Full builds that utilize external dependencies, also require setting +certain environment variables as discussed below. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. There are two primary -methods of building the code `cmake` and `gmake` which are provided +Instructions on building the necessary dependencies are below +and should be successfully installed first. There are two primary +methods of building MFIX-Exa: `cmake` and `gmake` which are provided separately below. cmake ~~~~~ .. tabs:: - + .. tab:: CPU .. code:: bash @@ -194,8 +194,8 @@ cmake Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. Set environment helpers @@ -204,7 +204,7 @@ prior to following any of the full build instructions above. export CC=$(which cc) export CXX=$(which CC) - mkdir $HOME/scratch && cd $HOME/scratch + mkdir $HOME/scratch && cd $HOME/scratch #. HYPRE @@ -387,8 +387,8 @@ Common Slurm commands: Example run scripts: -Example run script for GPU is below. For CPU-only, -remove ``module load nvhpc-mixed`` and ``export MPICH_GPU_SUPPORT_ENABLED=1``, +Example run script for GPU is below. For CPU-only, +remove ``module load nvhpc-mixed`` and ``export MPICH_GPU_SUPPORT_ENABLED=1``, and the options ``ppn`` and ``--cpu-bind core`` from the run line. .. code:: bash diff --git a/docs/source_docs/references/hpc/Summit.rst b/docs/source_docs/references/hpc/Summit.rst index 808f752d316816a448ef8e80816921e596058047..20b05e964c46ad2275fae99a4e248bbaa23c6b07 100644 --- a/docs/source_docs/references/hpc/Summit.rst +++ b/docs/source_docs/references/hpc/Summit.rst @@ -1,20 +1,20 @@ OLCF Summit =========== -If this is your first time building MFIX-Exa on OLCF's Summit, please +If this is your first time building MFIX-Exa on OLCF's Summit, please review the general notes below and `Basics`_ section first. * To access Summit, you will need an `OLCF account `_ - and an RSA token. + and an RSA token. * The name of the MFIX-Exa account is ``cfd122`` -* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a ``build`` directory. - The gmake instructions compile to a ``exec`` directory. -* For the dependencies, it is assumed that you have set the +* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a ``build`` directory. + The gmake instructions compile to a ``exec`` directory. +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -24,11 +24,11 @@ review the general notes below and `Basics`_ section first. export CSG_LIB_DIR=$HOME/ export ASCENT_INSTALL_DIR=$HOME/ - to a path that you have read/write access to. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. -* After building the ``mfix`` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command + to a path that you have read/write access to. + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. +* After building the ``mfix`` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the ``build`` directory .. code:: bash @@ -41,51 +41,51 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -All of the build instructions below have been tested with the +All of the build instructions below have been tested with the following modules -.. code:: bash +.. code:: bash module load cmake/3.23.2 module load gcc/9.3.0 -Do **NOT** purge your default modules. You do not need to load +Do **NOT** purge your default modules. You do not need to load an MPI module, this is handled by the preloaded ``spectrum-mpi`` module. The GPU-enabled builds additionally require -.. code:: bash +.. code:: bash module load cuda/11.5.2 -Full builds that utilize external dependencies, also require setting -certain environment variables as discussed below. +Full builds that utilize external dependencies, also require setting +certain environment variables as discussed below. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. There are two primary -methods of building the code ``cmake`` and ``gmake`` which are provided +Instructions on building the necessary dependencies are below +and should be successfully installed first. There are two primary +methods of building MFIX-Exa: ``cmake`` and ``gmake`` which are provided separately below. cmake ~~~~~ .. tabs:: - + .. tab:: CPU .. code:: bash @@ -202,9 +202,9 @@ cmake gmake ~~~~~ - + .. tabs:: - + .. tab:: CPU .. code:: bash @@ -218,13 +218,13 @@ gmake USE_CSG=FALSE \ USE_HYPRE=FALSE \ DEBUG=FALSE - + .. tab:: GPU .. code:: bash - - make -C exec -j8 + + make -C exec -j8 COMP=gnu \ USE_MPI=TRUE \ USE_OMP=FALSE \ @@ -268,7 +268,7 @@ gmake .. tab:: GPU-full .. code:: bash - + module load mpfr/4.0.2 module load boost/1.77.0 @@ -299,8 +299,8 @@ gmake Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. Set environment helpers @@ -309,14 +309,14 @@ prior to following any of the full build instructions above. export CC=$(which gcc) export CXX=$(which g++) export FC=$(which gfortran) - mkdir $HOME/scratch && cd $HOME/scratch + mkdir $HOME/scratch && cd $HOME/scratch + +.. note:: -.. note:: - - For whatever reason, building hypre with GPU support does not like this. - So if you are building everything all at once, start with hypre with cuda + For whatever reason, building hypre with GPU support does not like this. + So if you are building everything all at once, start with hypre with cuda and then set your compilers and build the rest (including hypre without cuda). - + #. HYPRE @@ -330,7 +330,7 @@ prior to following any of the full build instructions above. pushd hypre/src/ git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install + make -j8 install popd .. tab:: GPU @@ -351,7 +351,7 @@ prior to following any of the full build instructions above. --with-cuda-home=$OLCF_CUDA_ROOT \ --enable-cusparse \ --enable-curand - make -j8 install + make -j8 install popd #. Catch2 @@ -399,7 +399,7 @@ prior to following any of the full build instructions above. make -j8 install popd -#. CSG EB library (**gmake**) +#. CSG EB library (**gmake**) For the gmake install instructions, you need to install ``libcsgeb`` to ``$CSG_LIB_DIR`` using either cmake or gmake: @@ -503,11 +503,11 @@ Common Slurm commands: * ``bsub runit_cpu.sh`` submit a cpu job to the queue * ``bjobs -u USER`` check job status of user USER * ``bkill JOBID`` kill a job with id JOBID -* ``bsub -W 0:20 -nnodes 1 -P `` grab a node interactively for 20 minutes +* ``bsub -W 0:20 -nnodes 1 -P `` grab a node interactively for 20 minutes -Example run scripts for GPU is below. For CPU-only, -you remove ``module load cuda``, set ``--gpu_per_rs 0`` and remove ``--bind packed:1`` -from the run line. +Example run scripts for GPU is below. For CPU-only, +you remove ``module load cuda``, set ``--gpu_per_rs 0`` and remove ``--bind packed:1`` +from the run line. .. code:: bash @@ -526,7 +526,5 @@ from the run line. set -x omp=1 export OMP_NUM_THREADS=${omp} - - jsrun --nrs 12 --tasks_per_rs 1 --cpu_per_rs 1 --gpu_per_rs 1 --launch_distribution packed --bind packed:1 ./mfix inputs > screen.txt - + jsrun --nrs 12 --tasks_per_rs 1 --cpu_per_rs 1 --gpu_per_rs 1 --launch_distribution packed --bind packed:1 ./mfix inputs > screen.txt diff --git a/docs/source_docs/references/size_distributions.rst b/docs/source_docs/references/size_distributions.rst index 4400978b41ce3cbf08ab33afc82f2f8f69109327..649c7b9a1c72dcabba767c9e465a5e8fd1c69cb3 100644 --- a/docs/source_docs/references/size_distributions.rst +++ b/docs/source_docs/references/size_distributions.rst @@ -27,7 +27,7 @@ Volume-weighted diameter distribution Number-weighted and volume-weighted distributions are not interchangeable. Failing to specify the correct distribution weighting can result in unexpected behavior. -The code uses both distribution weightings, computing (or approximating) +MFIX uses both distribution weightings, computing (or approximating) whichever distribution weighting that was not provided. For example, if a number-weighted normal distribution is provided, the corresponding volume-weighted normal distribution is computed. The volume-weighted distribution is used to calculate diff --git a/docs/source_docs/references/units.rst b/docs/source_docs/references/units.rst index 605f45e1d5be4c9bde44198b6e68f7d7b7fdbad9..5ece703964824a1be37f0c5e0bcbcf987884eb25 100644 --- a/docs/source_docs/references/units.rst +++ b/docs/source_docs/references/units.rst @@ -3,7 +3,7 @@ System of Units MFIX-Exa adopts the International System of Units (SI). Simulations have to be set up accordingly. In the following table we provide a list of some of the -physical quantities we can specify in the input file and their correspondent +physical quantities we can specify in the input file and their corresponding units. +--------------------------------+-----------------------------------------+ @@ -49,5 +49,3 @@ units. +--------------------------------+-----------------------------------------+ | reaction rate (Euler-Lagrange) | [:math:`mol \cdot s^{-1}`] | +--------------------------------+-----------------------------------------+ - - diff --git a/docs/source_docs/user_guide/inputs/boundary_conditions.rst b/docs/source_docs/user_guide/inputs/boundary_conditions.rst index d2790de5d67827f50c78e4e39f652d83dbef4016..975af06b6e2ff5da82b0496d4ed8b53bbb00f712 100644 --- a/docs/source_docs/user_guide/inputs/boundary_conditions.rst +++ b/docs/source_docs/user_guide/inputs/boundary_conditions.rst @@ -163,33 +163,33 @@ conditions. Each entry must be preceded by the prefix ``bc.[region_name]``. Like inflows, the fluid temperature, pressure, and species composition must be provided when appropriate. -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| | Description | Type | Default | -+=====================+=======================================================================+=============+===========+ -| fluid.velocity | [Required if not `volflow` or `massflow`] | | | -| | Inflow fluid velocity on EB faces | Reals | 0 0 0 | -| | contained in the (tridimensional) region. | | | -| | Note that if only one value is specified, that is assumed to | | | -| | be the magnitude in the direction of the EB face's normal. | | | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| fluid.volflow | [Required if not `velocity` or `massflow`] | | | -| | Inflow BC for fluid volumetric flow | Real | 0 | -| | rate in the (tridimensional) region. The flow is assumed to be | | | -| | normal to the EB surface in the region. | | | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| fluid.massflow | [Required if not `velocity` or `volflow`] | | | -| | Inflow BC for fluid mass flow rate in the (tridimensional) region. | Real | 0 | -| | The flow is assumed to be normal to the EB surface in the region. | | | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| fluid.volfrac | [Required] Volume fraction. | Real | 0 | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| eb.normal | [Optional] When specified, only cells with EB face normal that is | Reals | 0 0 0 | -| | parallel and opposite in direction to the specified values | | | -| | are imposed with the inflow velocity. | | | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ -| eb.normal_tol | [Optional] Used in conjunction with `eb.normal`. It determines the | Real | 0 | -| | tolerance (in degrees) for choosing cells with a specific normal. | | | -+---------------------+-----------------------------------------------------------------------+-------------+-----------+ ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| | Description | Type | Default | ++=======================+=======================================================================+=============+===========+ +| [fluid_name].velocity | [Required if not `volflow` or `massflow`] | | | +| | Inflow fluid velocity on EB faces | Reals | 0 0 0 | +| | contained in the (tridimensional) region. | | | +| | Note that if only one value is specified, that is assumed to | | | +| | be the magnitude in the direction of the EB face's normal. | | | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| [fluid_name].volflow | [Required if not `velocity` or `massflow`] | | | +| | Inflow BC for fluid volumetric flow | Real | 0 | +| | rate in the (tridimensional) region. The flow is assumed to be | | | +| | normal to the EB surface in the region. | | | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| [fluid_name].massflow | [Required if not `velocity` or `volflow`] | | | +| | Inflow BC for fluid mass flow rate in the (tridimensional) region. | Real | 0 | +| | The flow is assumed to be normal to the EB surface in the region. | | | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| [fluid_name].volfrac | [Required] Volume fraction. | Real | 0 | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| eb.normal | [Optional] When specified, only cells with EB face normal that is | Reals | 0 0 0 | +| | parallel and opposite in direction to the specified values | | | +| | are imposed with the inflow velocity. | | | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| eb.normal_tol | [Optional] Used in conjunction with `eb.normal`. It determines the | Real | 0 | +| | tolerance (in degrees) for choosing cells with a specific normal. | | | ++-----------------------+-----------------------------------------------------------------------+-------------+-----------+ Note that only one of ``velocity`` or ``volflow`` or ``massflow`` should be specified. diff --git a/docs/source_docs/user_guide/inputs/fluid_model.rst b/docs/source_docs/user_guide/inputs/fluid_model.rst index 312fb52ecc59a58739ab3e259cb56354952fafff..9db1764d6dd78741c8686ee68feda59c97e35532 100644 --- a/docs/source_docs/user_guide/inputs/fluid_model.rst +++ b/docs/source_docs/user_guide/inputs/fluid_model.rst @@ -8,7 +8,7 @@ The following inputs are defined using the prefix ``fluid``: +--------------------------------------------+-------------------------------------------------------------+--------+----------+ | | Description | Type | Default | +============================================+=============================================================+========+==========+ -| solve | Specify the name of the fluid or set to ``None`` to disable | String | ``None`` | +| solve | Specify the name of the fluid or set to ``None`` to disable | String | "None" | | | the fluid solver. The name assigned to the fluid solver is | | | | | used to specify fluid properties and initial and boundary | | | | | conditions. It is common to use the name ``fluid``. | | | diff --git a/docs/source_docs/user_guide/inputs/model_options.rst b/docs/source_docs/user_guide/inputs/model_options.rst index 3f09d1ed4affc63f70fcb2241a573e3985c187b3..8f5e2de5f43bbadd0688511f5d9f0a8d38c3e4d5 100644 --- a/docs/source_docs/user_guide/inputs/model_options.rst +++ b/docs/source_docs/user_guide/inputs/model_options.rst @@ -354,7 +354,7 @@ The following input is defined using the prefix ``mfix.drag``: The ``UserDrag`` keyword is used to invoke a user-defined drag model. This is accomplished by copying ``src/usr/usr_drag.cpp`` file into the build directory, implementing the desired - drag model, and recompiling the code. An example can be found in ``tests/DEM06-x``. + drag model, and recompiling MFIX-Exa. An example can be found in ``tests/DEM06-x``. Heat transfer coefficients diff --git a/docs/source_docs/user_guide/inputs/multigrid_solvers.rst b/docs/source_docs/user_guide/inputs/multigrid_solvers.rst index adfbfaa29fa9f4c3d7821255db8fc2d3a6b689bc..682b6ac65d7ee604276fd4cea85cfd39f435688d 100644 --- a/docs/source_docs/user_guide/inputs/multigrid_solvers.rst +++ b/docs/source_docs/user_guide/inputs/multigrid_solvers.rst @@ -141,20 +141,20 @@ The following inputs are defined using the prefix ``diffusion``: Hypre Settings -------------- -`hypre` settings are specified using the following inputs which are read directly by AMReX when `hypre` is used as +``hypre`` settings are specified using the following inputs which are read directly by AMReX when ``hypre`` is used as the bottom solver for the MAC and/or nodal projections. By default, these inputs are defined using the prefix ``hypre``, however different settings for the nodal and MAC projections can be used by specifying a ``hypre_namespace`` for each solver. .. warning:: - * MFIX-Exa must be built with `hypre` enabled to use `hypre` as the bottom solver. - * If `hypre` is used for the nodal and MAC projections, then either the ``hypre_namespace`` must be left as the default (``hypre``) + * MFIX-Exa must be built with ``hypre`` enabled to use ``hypre`` as the bottom solver. + * If ``hypre`` is used for the nodal and MAC projections, then either the ``hypre_namespace`` must be left as the default (``hypre``) or unique name spaces must be provided for both. * These inputs are all case-sensitive. -These inputs follow the typical approach for using `hypre` in MFIX-Exa where `BoomerAMG` is employed as a preconditioner +These inputs follow the typical approach for using ``hypre`` in MFIX-Exa where ``BoomerAMG`` is employed as a preconditioner and `GMRES` for the linear solver. +-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+ diff --git a/docs/source_docs/user_guide/inputs/solids_model.rst b/docs/source_docs/user_guide/inputs/solids_model.rst index ecab98cf99b20f6bdbc29a5705d88f9432bc71db..1058b19737eebbefb753ef2fa38d374b3ea90a08 100644 --- a/docs/source_docs/user_guide/inputs/solids_model.rst +++ b/docs/source_docs/user_guide/inputs/solids_model.rst @@ -150,7 +150,7 @@ Enabling the DEM solver and specifying model options. The following keys must b | | |RollingFricModelB| | | | | | | | | +-------------------------------+-------------------------------------------------------------------------+----------+-----------+ -| rolling_friction_coefficient | Rolling friction coefficient when using a rolling friction model | Real | 0 | +| rolling_friction.coefficient | Rolling friction coefficient when using a rolling friction model | Real | 0 | +-------------------------------+-------------------------------------------------------------------------+----------+-----------+ diff --git a/docs/source_docs/user_guide/quick_start/FirstSimulation.rst b/docs/source_docs/user_guide/quick_start/FirstSimulation.rst index 4a4a02e93b78e579ca679f62c9550faeac389bed..fafbd2ee400591bcc21aee82136eb1088a85cb44 100644 --- a/docs/source_docs/user_guide/quick_start/FirstSimulation.rst +++ b/docs/source_docs/user_guide/quick_start/FirstSimulation.rst @@ -1,7 +1,7 @@ First Simulation ================ -Now that you have compiled the code and have an executable, +Now that you have compiled MFIX-Exa and have an executable, we will run a simple example that contains fluid, particles, and embedded boundaries. This particular example is particles flowing down a cylinder. @@ -15,6 +15,6 @@ Then, run the example with: >> path/to/mfix/benchmarks/05-cyl-fluidbed/Size0001/inputs -The code will generate plotfiles with prefix ``plt*`` and geometry files +MFIX-Exa code will generate plotfiles with prefix ``plt*`` and geometry files with prefix ``eb*``. See :ref:`Chap:Visualization` for how to view this dataset. diff --git a/duplicate-keys.txt b/duplicate-keys.txt new file mode 100644 index 0000000000000000000000000000000000000000..efa155e121adf2997a2b34813017ba7b2a6ec497 --- /dev/null +++ b/duplicate-keys.txt @@ -0,0 +1,11 @@ +Duplicate keys: +ascent.actions: ascent.rst, plotting.rst +bc.[region_name].[fluid_name].massflow: boundary_conditions.rst, boundary_conditions.rst +bc.[region_name].[fluid_name].velocity: boundary_conditions.rst, boundary_conditions.rst +bc.[region_name].[fluid_name].volflow: boundary_conditions.rst, boundary_conditions.rst +bc.[region_name].[fluid_name].volfrac: boundary_conditions.rst, boundary_conditions.rst +hypre.bamg_num_down_sweeps: multigrid_solvers.rst, multigrid_solvers.rst +hypre.bamg_num_sweeps: multigrid_solvers.rst, multigrid_solvers.rst +mfix.ascent_int: ascent.rst, plotting.rst +mfix.ascent_per_approx: ascent.rst, plotting.rst +mfix.restart: checkpointing.rst, initialization.rst diff --git a/nonexistent-keys.txt b/nonexistent-keys.txt new file mode 100644 index 0000000000000000000000000000000000000000..329d82cc95cf81f7883115f13c1d925ba4ef93c4 --- /dev/null +++ b/nonexistent-keys.txt @@ -0,0 +1,31 @@ +csg.geometry_filename +ic.[region_name].[solid_name].bins +ic.[region_name].[solid_name].type +mac_proj.bottom_maxiter +mac_proj.bottom_verbose +mac_proj.maxiter +mac_proj.verbose +mfix.deposition_diffusion_coeff +mfix.godunov_ppm +mfix.plt_T_p +mfix.plt_drag_p +mfix.plt_omega_p +mfix.plt_phase +mfix.plt_radius +mfix.plt_ro_p +mfix.plt_statwt +mfix.plt_vel_p +mfix.solids.[region_name].plt_cp_s +mfix.solids.[region_name].plt_h_s_txfr +mfix.solids.[region_name].plt_mass +mfix.solids.[region_name].plt_mass_sn_txfr +mfix.solids.[region_name].plt_omoi +mfix.solids.[region_name].plt_vel_s_txfr +mfix.solids.[region_name].plt_volume +mfix.use_drag_coeff_in_proj_gp +mfix.use_drag_in_godunov +nodal_proj.bottom_maxiter +nodal_proj.bottom_verbose +nodal_proj.maxiter +nodal_proj.verbose +stl.geometry_filename \ No newline at end of file diff --git a/undocumented-keys.txt b/undocumented-keys.txt new file mode 100644 index 0000000000000000000000000000000000000000..165d31ffe0f166d25997a24de61f2a917c69bf46 --- /dev/null +++ b/undocumented-keys.txt @@ -0,0 +1,114 @@ +[fluid_name].trac0 +amr.restart +bc.[region_name].[solid_name].density +bc.[region_name].[solid_name].density.constant +bc.[region_name].[solid_name].species.[species_name] +bc.[region_name].[solid_name].temperature +bc.[region_name].[solid_name].velocity +bc.[region_name].[solid_name].volflow +bc.[region_name].[solid_name].volfrac +bc.[region_name].solids +catalyst.catalyst_on_restart +catalyst.enabled +catalyst.implementation +catalyst.library_path +catalyst.script +chemistry.solids.density +dem.PolyNeighSearch +dem.PolyNumTypes +dem.PolyRefRatios +dem.cg_particles_per_cell_at_pack +dem.coarse_grain +diffusion.agg_grid_size +diffusion.atol +diffusion.hypre_interface +diffusion.hypre_namespace +diffusion.mg_max_fmg_iter +diffusion.rtol +diffusion.verbose_solver +eb2.extend_domain_face +eb2.small_volfrac +fab.format +fluid.thermal_conductivity.constant +hopper.center +hopper.direction +hopper.funnel_height +hopper.funnel_radius +hopper.orifice_radius +ic.[region_name].[fluid_name].pressure +ic.[region_name].[solid_name].[property_name].bins +ic.[region_name].[solid_name].[property_name].interpolate +ic.[region_name].[solid_name].[property_name].type +ic.[region_name].granular_temperature +mfix.advect_momentum +mfix.advect_tracer +mfix.agg_grid_size +mfix.ascent_on_restart +mfix.checkpoint_files_output +mfix.constraint.include_depdt +mfix.deposition.filter +mfix.deposition.filter.constant +mfix.deposition.filter.variable +mfix.deposition.filter.variable.min_eps +mfix.dual_grid +mfix.geom_chk_ccse_regtest +mfix.geometry_filename +mfix.monitors.[monitor_name].count +mfix.overstep_end_time +mfix.particle_sorting_bin +mfix.particles.enthalpy_source +mfix.particles.update_enthalpy +mfix.particles.update_mass +mfix.particles.update_momentum +mfix.plt_D_g +mfix.plt_MW_g +mfix.plt_X_g +mfix.plt_chem_txfr +mfix.plt_cp_g +mfix.plt_cp_gk +mfix.plt_h_g +mfix.plt_h_gk +mfix.plt_k_g +mfix.plt_proc +mfix.plt_proc_p +mfix.plt_trac +mfix.plt_txfr +mfix.redistribute_after_initial_nodal_proj +mfix.reports.mass_balance_int +mfix.reports.mass_balance_per_approx +mfix.solids.[region_name].plt_acceleration +mfix.solids.[region_name].plt_ep_s +mfix.solids.[region_name].plt_fluid_vars +mfix.solids.[region_name].plt_pft_neighbor +mfix.solids.[region_name].plt_pft_neighbor_flags +mfix.solids.[region_name].plt_ptype +mfix.solids.[region_name].plt_vm_coeff +mfix.sort_particle_int +mfix.stop_for_unused_inputs +mfix.tag.grad_rho +mfix.tag.regions +mfix.tag.rho +mfix.tag.vorticity +mfix.test_tracer_conservation +mfix.use_new_depdt_algo +mfix.use_ppm +particles.reduceGhostParticles +pic.advance_vel_p +pic.beta +pic.close_pack +pic.damping_factor +pic.damping_factor_wall_normal +pic.damping_factor_wall_tangent +pic.initial_step_type +pic.max_iter +pic.parcels_per_cell_at_pack +pic.pressure_coefficient +pic.small_number +pic.solve +pic.velocity_reference_frame +pic.verbose +pic2dem.convert +stl.internal_flow +tracer +tracer.diff_coeff +viscosity.molecular \ No newline at end of file