From 065ff57790021bcbf36f3e49c58cf353132b37d5 Mon Sep 17 00:00:00 2001 From: Deepak Rangarajan Date: Thu, 16 Feb 2023 18:28:53 +0000 Subject: [PATCH] HPC specific build instructions --- .github/workflows/build.yml | 2 +- README.md | 4 +- docs/source_docs/GettingStarted_Chapter.rst | 5 +- docs/source_docs/conf.py | 3 +- .../getting_started/BuildingGMake.rst | 66 -- .../{BuildingCMake.rst => BuildingMFIX.rst} | 199 +++--- docs/source_docs/getting_started/Hpc.rst | 25 + .../source_docs/getting_started/Structure.rst | 22 - docs/source_docs/getting_started/hpc/Cori.rst | 4 + .../getting_started/hpc/Crusher.rst | 4 + .../getting_started/hpc/Joule2.rst | 597 ++++++++++++++++++ .../getting_started/hpc/Perlmutter.rst | 4 + .../getting_started/hpc/Polaris.rst | 4 + .../source_docs/getting_started/hpc/Spock.rst | 4 + .../getting_started/hpc/Summit.rst | 4 + 15 files changed, 738 insertions(+), 209 deletions(-) delete mode 100644 docs/source_docs/getting_started/BuildingGMake.rst rename docs/source_docs/getting_started/{BuildingCMake.rst => BuildingMFIX.rst} (72%) create mode 100644 docs/source_docs/getting_started/Hpc.rst delete mode 100644 docs/source_docs/getting_started/Structure.rst create mode 100644 docs/source_docs/getting_started/hpc/Cori.rst create mode 100644 docs/source_docs/getting_started/hpc/Crusher.rst create mode 100644 docs/source_docs/getting_started/hpc/Joule2.rst create mode 100644 docs/source_docs/getting_started/hpc/Perlmutter.rst create mode 100644 docs/source_docs/getting_started/hpc/Polaris.rst create mode 100644 docs/source_docs/getting_started/hpc/Spock.rst create mode 100644 docs/source_docs/getting_started/hpc/Summit.rst diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index c2e9613..6b0bdf0 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -20,7 +20,7 @@ jobs: run: sudo apt install doxygen - name: Install Sphinx - run: pip install sphinx sphinx-rtd-theme + run: pip install sphinx sphinx-rtd-theme sphinx-tabs - name: Checkout code uses: actions/checkout@v2 diff --git a/README.md b/README.md index 542d4ce..0b2efbd 100644 --- a/README.md +++ b/README.md @@ -21,14 +21,14 @@ can be built locally by: 1. Install sphinx: ```shell -> pip install sphinx sphinx_rtd_theme +> pip install sphinx sphinx_rtd_theme sphinx-tabs ``` 2. Build the documentation: ```shell > mkdir build -> sphinx-build -b html docs/source/ build +> sphinx-build -b html docs/source_docs/ build ``` The HTML pages will now be located in the `build` directory. Open the diff --git a/docs/source_docs/GettingStarted_Chapter.rst b/docs/source_docs/GettingStarted_Chapter.rst index bd4bbb3..030b653 100644 --- a/docs/source_docs/GettingStarted_Chapter.rst +++ b/docs/source_docs/GettingStarted_Chapter.rst @@ -27,8 +27,7 @@ the simulations results. .. toctree:: :maxdepth: 1 - Source directory overview - Compiling with gmake - Compiling with cmake + Building MFIX-Exa Running your first simulation Visualizing simulation results + HPC diff --git a/docs/source_docs/conf.py b/docs/source_docs/conf.py index 8b1d0a3..ee9995e 100644 --- a/docs/source_docs/conf.py +++ b/docs/source_docs/conf.py @@ -33,7 +33,8 @@ import sphinx_rtd_theme # ones. extensions = ['sphinx.ext.mathjax', 'sphinx.ext.githubpages', - 'sphinx.ext.viewcode'] + 'sphinx.ext.viewcode', + 'sphinx_tabs.tabs'] # Add any paths that contain templates here, relative to this directory. templates_path = ['ytemplates'] diff --git a/docs/source_docs/getting_started/BuildingGMake.rst b/docs/source_docs/getting_started/BuildingGMake.rst deleted file mode 100644 index 3d66457..0000000 --- a/docs/source_docs/getting_started/BuildingGMake.rst +++ /dev/null @@ -1,66 +0,0 @@ -Building MFIX-Exa with gmake -============================ - -If you want to use gmake to build MFIX_Exa, you will need to -clone amrex and AMReX-Hydro into a local directories, and also -clone mfix: - -.. code:: shell - - > git clone https://github.com/AMReX-Codes/amrex.git - > git clone https://github.com/AMReX-Codes/AMReX-Hydro.git - > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git - > cd mfix/exec - -Then, edit the GNUmakefile (or set an environment variable) -to define AMREX_HOME and AMREX_HYDRO_HOME -to be the path to the directories where you have put amrex -and AMReX-Hydro. Note that the default locations of -AMREX_HOME and AMREX_HYDRO_HOME are in the subprojects directory, -which is where a cmake configuration may clone these repositories. -Other options that you can set include - -+-----------------+----------------------------------+------------------+-------------+ -| Option name | Description | Possible values | Default | -| | | | value | -+=================+==================================+==================+=============+ -| COMP | Compiler (gnu or intel) | gnu / intel | gnu | -+-----------------+----------------------------------+------------------+-------------+ -| USE_MPI | Enable MPI | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_OMP | Enable OpenMP | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_CSG | Enable CSG geometry file support | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_CUDA | Enable CUDA GPU support | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_HIP | Enable HIP GPU support | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_DPCPP | Enable DPCPP GPU support | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| USE_HYPRE | Enable HYPRE support | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| DEBUG | Create a DEBUG executable | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| PROFILE | Include profiling info | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| TRACE_PROFILE | Include trace profiling info | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| COMM_PROFILE | Include comm profiling info | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| TINY_PROFILE | Include tiny profiling info | TRUE / FALSE | FALSE | -+-----------------+----------------------------------+------------------+-------------+ -| DIM | Dimensionality of problem | 2 / 3 | 3 | -+-----------------+----------------------------------+------------------+-------------+ - -.. note:: - **Do not set both USE_OMP and USE_CUDA/HIP/DPCPP to true.** - -Then type - -.. code:: shell - - > make -j - -An executable will appear; the executable name will reflect -some of the build options above. diff --git a/docs/source_docs/getting_started/BuildingCMake.rst b/docs/source_docs/getting_started/BuildingMFIX.rst similarity index 72% rename from docs/source_docs/getting_started/BuildingCMake.rst rename to docs/source_docs/getting_started/BuildingMFIX.rst index db9941e..9bb7a96 100644 --- a/docs/source_docs/getting_started/BuildingCMake.rst +++ b/docs/source_docs/getting_started/BuildingMFIX.rst @@ -1,5 +1,80 @@ -Building MFIX-Exa with cmake -============================ +Building MFIX-Exa +================== + +This page gives the generic instructions for building MFIX-Exa using gmake and cmake. +For HPC cluser specific instructions, please refer to :ref:`GettingStarted:HPC`. + + +Building with gmake +-------------------- + +If you want to use gmake to build MFIX_Exa, you will need to +clone amrex and AMReX-Hydro into a local directories, and also +clone mfix: + +.. code:: shell + + > git clone https://github.com/AMReX-Codes/amrex.git + > git clone https://github.com/AMReX-Codes/AMReX-Hydro.git + > git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git + > cd mfix/exec + +Then, edit the GNUmakefile (or set an environment variable) +to define AMREX_HOME and AMREX_HYDRO_HOME +to be the path to the directories where you have put amrex +and AMReX-Hydro. Note that the default locations of +AMREX_HOME and AMREX_HYDRO_HOME are in the subprojects directory, +which is where a cmake configuration may clone these repositories. +Other options that you can set include + ++-----------------+----------------------------------+------------------+-------------+ +| Option name | Description | Possible values | Default | +| | | | value | ++=================+==================================+==================+=============+ +| COMP | Compiler (gnu or intel) | gnu / intel | gnu | ++-----------------+----------------------------------+------------------+-------------+ +| USE_MPI | Enable MPI | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_OMP | Enable OpenMP | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_CSG | Enable CSG geometry file support | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_CUDA | Enable CUDA GPU support | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_HIP | Enable HIP GPU support | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_DPCPP | Enable DPCPP GPU support | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| USE_HYPRE | Enable HYPRE support | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| DEBUG | Create a DEBUG executable | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| PROFILE | Include profiling info | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| TRACE_PROFILE | Include trace profiling info | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| COMM_PROFILE | Include comm profiling info | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| TINY_PROFILE | Include tiny profiling info | TRUE / FALSE | FALSE | ++-----------------+----------------------------------+------------------+-------------+ +| DIM | Dimensionality of problem | 2 / 3 | 3 | ++-----------------+----------------------------------+------------------+-------------+ + +.. note:: + **Do not set both USE_OMP and USE_CUDA/HIP/DPCPP to true.** + +Then type + +.. code:: shell + + > make -j + +An executable will appear; the executable name will reflect +some of the build options above. + + +Building with cmake +-------------------- CMake build is a two-step process. First ``cmake`` is invoked to create configuration files and makefiles in a chosen directory (``builddir``). @@ -44,7 +119,7 @@ configuration options must meet MFIX-Exa requirements. .. _sec:build:superbuild: SUPERBUILD Instructions (recommended) -------------------------------------- +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< By default MFIX-Exa CMake looks for an existing AMReX installation on the system. If none is found, it falls back to **SUPERBUILD** mode. In this mode, MFIX-Exa CMake inherents AMReX CMake targets and configuration options, that is, MFIX-Exa and AMReX are configured and built as a single entity @@ -74,7 +149,7 @@ For example, to enable AMReX profiling capabilities in MFIX_Exa, configure as fo Working with the AMReX submodule -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **SUPERBUILD** mode relies on a git submodule to checkout the AMReX git repository. If the AMReX submodule is not initialized, **SUPERBUILD** mode will initialize it and checkout @@ -133,11 +208,11 @@ subprojects/amrex``.) .. _sec:build:standalone: -**STANDALONE** instructions ---------------------------------------------------------------------- +STANDALONE instructions +<<<<<<<<<<<<<<<<<<<<<<< Building AMReX -~~~~~~~~~~~~~~~~~~~ +>>>>>>>>>>>>>> Clone AMReX from the official Git repository. Note that the only branch available is *development*: @@ -160,7 +235,7 @@ The options **AMReX\_PARTICLES=yes**, **AMReX\_EB=yes** and **AMReX\_PLOTFILE\_ Building MFIX-Exa -~~~~~~~~~~~~~~~~~ +>>>>>>>>>>>>>>>>> Clone and build MFIX-Exa: @@ -220,8 +295,8 @@ for an available AMReX installation. -Few more notes on building MFIX-Exa ------------------------------------ +Few more notes on building MFIX-Exa with cmake +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< The system defaults compilers can be overwritten as follows: @@ -245,107 +320,3 @@ For GPU builds, MFIX-Exa relies on the `AMReX GPU build infrastructure ``, or by defining the *environmental variable* ``AMREX_CUDA_ARCH`` (all caps). If no GPU architecture is specified, CMake will try to determine which GPU is supported by the system. - - -Building MFIX-Exa for Cori (NERSC) ------------------------------------ - -Standard build -~~~~~~~~~~~~~~~~~~~ - -For the Cori cluster at NERSC, you first need to load/unload modules required to build MFIX-Exa. - -.. code:: shell - - > module unload altd - > module unload darshan - > module load cmake/3.14.0 - -The default options for Cori are the **Haswell** architecture and **Intel** compiler, if you want to compile with the **Knight's Landing (KNL)** architecture: - -.. code:: shell - - > module swap craype-haswell craype-mic-knl - -Or use the **GNU** compiler: - -.. code:: shell - - > module swap PrgEnv-intel PrgEnv-gnu - -Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`. - -.. note:: - - The load/unload modules options could be saved in the `~/.bash_profile.ext` - - -GPU build -~~~~~~~~~~~~~~~~~~~ - -To compile on the GPU nodes in Cori, you first need to purge your modules, most of which won't work on the GPU nodes - -.. code:: shell - - > module purge - -Then, you need to load the following modules: - -.. code:: shell - - > module load modules esslurm gcc cuda openmpi/3.1.0-ucx cmake/3.14.0 - -Currently, you need to use OpenMPI; mvapich2 seems not to work. - -Then, you need to use slurm to request access to a GPU node: - -.. code:: shell - - > salloc -N 1 -t 02:00:00 -c 80 -C gpu -A m1759 --gres=gpu:8 --exclusive - -This reservers an entire GPU node for your job. Note that you can’t cross-compile for the GPU nodes - you have to log on to one and then build your software. - -Finally, navigate to the base of the MFIX-Exa repository and compile in GPU mode: - -.. code:: shell - - > cd mfix - > mdkir build - > cd build - > cmake -DMFIX_GPU_BACKEND=CUDA -DAMReX_CUDA_ARCH=Volta -DCMAKE_CXX_COMPILER=g++ .. - > make -j - -For more information about GPU nodes in Cori -- ``_ - -Building MFIX-Exa for Summit (OLCF) ------------------------------------ - -For the Summit cluster at OLCF, you first need to load/unload modules required to build MFIX-Exa. - -.. code:: shell - - > module unload xalt - > module unload darshan - > module load gcc - > module load cmake/3.14.0 - -Now MFIX-Exa can be built following the :ref:`sec:build:superbuild`. - -To build MFIX-Exa for GPUs, you need to load cuda module: - -.. code:: shell - - > module load cuda/10.1.105 - -To compile: - -.. code:: shell - - > cd mfix - > mdkir build - > cd build - > cmake -DCMAKE_CXX_COMPILER=g++ -DMFIX_GPU_BACKEND=[NONE|CUDA] - > make -j - -An example of a *submission_script* for GPUs can be found in the repo ``mfix/tests/GPU_test/script.sh``. -For more information about Summit cluster: ``_ diff --git a/docs/source_docs/getting_started/Hpc.rst b/docs/source_docs/getting_started/Hpc.rst new file mode 100644 index 0000000..0555aad --- /dev/null +++ b/docs/source_docs/getting_started/Hpc.rst @@ -0,0 +1,25 @@ +.. _GettingStarted:HPC: + +Building on HPC Sytems +====================== + +On selected high-performance computing (HPC) systems MFIX-Exa has specific building instructions. +Follow the guide here instead of the generic instructions. + + +.. toctree:: + :caption: NETL Systems + :maxdepth: 1 + + hpc/Joule2 + +.. toctree:: + :caption: LCF Systems + :maxdepth: 1 + + hpc/Cori + hpc/Crusher + hpc/Perlmutter + hpc/Polaris + hpc/Spock + hpc/Summit diff --git a/docs/source_docs/getting_started/Structure.rst b/docs/source_docs/getting_started/Structure.rst deleted file mode 100644 index 0980fdf..0000000 --- a/docs/source_docs/getting_started/Structure.rst +++ /dev/null @@ -1,22 +0,0 @@ -.. _Chap:Structure: - -Directory overview -================== - -+---------------+--------------------------------------------------+ -| File | Description | -+===============+==================================================+ -| benchmarks | UC Benchmark cases (see benchmark/README.md) | -+---------------+--------------------------------------------------+ -| exec | Directory for building with gmake | -+---------------+--------------------------------------------------+ -| src | Source files | -+---------------+--------------------------------------------------+ -| subprojects | Location of dependent git repos for cmake builds | -+---------------+--------------------------------------------------+ -| tests | Regression tests (see tests/README.md) | -+---------------+--------------------------------------------------+ -| tools | cmake configuration files | -+---------------+--------------------------------------------------+ -| tutorials | Additional test cases | -+---------------+--------------------------------------------------+ diff --git a/docs/source_docs/getting_started/hpc/Cori.rst b/docs/source_docs/getting_started/hpc/Cori.rst new file mode 100644 index 0000000..8ffe1f2 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Cori.rst @@ -0,0 +1,4 @@ +Summit +======= + +Coming soon! diff --git a/docs/source_docs/getting_started/hpc/Crusher.rst b/docs/source_docs/getting_started/hpc/Crusher.rst new file mode 100644 index 0000000..668710c --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Crusher.rst @@ -0,0 +1,4 @@ +Crusher +======== + +Coming soon! diff --git a/docs/source_docs/getting_started/hpc/Joule2.rst b/docs/source_docs/getting_started/hpc/Joule2.rst new file mode 100644 index 0000000..a38c012 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Joule2.rst @@ -0,0 +1,597 @@ +Joule 2.0 +========== + +Basics +------- + +Access +>>>>>>> + +You need an NETL HPC account, yubikey and the `HPC client `_ + + +Running Jobs +>>>>>>>>>>>> + +Common Slurm commands: + +* **sinfo** see available/allocated resources +* **sbatch runit_cpu.sh** submit a cpu job to the queue +* **squeue -u USER** check job status of user USER +* **squeue -p PARTITION** check job status of partition PARTITION +* **scancel JOBID** kill a job with id JOBID +* **salloc -N 1 -p gpu** grab a GPU node interactively (for up to 48 hrs) +* **salloc -N 2 -p dev -q dev** grab two development nodes (for up to 2 hrs) + + +Building MFIX-Exa: cmake +------------------------- + +Standalone Build +>>>>>>>>>>>>>>>>> + +#. Clone + + For the basic, standalone CPU and GPU builds, first clone the code, + checkout the desired branch, update the submodules and create a build directory + + .. code:: bash + + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + mkdir build && cd build/ + +#. Build mfix + + For a CPU load the necessary modules and build from the head node. For GPU first connect to an interactive node and then build: + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + ## build mfix + cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DMFIX_MPI=yes -DMFIX_OMP=no -DMFIX_GPU_BACKEND=NONE -DAMReX_TINY_PROFILE=no -DMFIX_CSG=no -DMFIX_HYPRE=no -DCMAKE_BUILD_TYPE=Release ../mfix + make -j 8 + + ## uncomment to build the pic2dem restarter app + #cmake -j8 --build . --target pic2dem + + .. tab:: GPU + + .. code:: bash + + ## connect to an interactive node + salloc -N 1 -p gpu + + ## load Modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load cuda/11.3 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + ## build mfix + cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DMFIX_MPI=yes -DMFIX_OMP=no -DMFIX_GPU_BACKEND=CUDA -DAMReX_TINY_PROFILE=no -DMFIX_CSG=no -DMFIX_HYPRE=no -DCMAKE_BUILD_TYPE=Release ../mfix && make -j 20 + + ## uncomment to build the pic2dem restarter app + #cmake -j8 --build . --target pic2dem + + + +Full Build +>>>>>>>>>>> + +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. + +Install dependencies +<<<<<<<<<<<<<<<<<<<<< + +#. Setup + + .. code:: bash + + ## create directories for installation + cd ~/packages/ + rm -rf old.mfix-exa_deps/ + mv mfix-exa_deps/ old.mfix-exa_deps/ + cd $HOME + rm -rf tmp.bld-deps/ + mkdir tmp.bld-deps && cd tmp.bld-deps/ + + ## set and init install directories + export MY_INSTALL_DIR=$(pwd)/mfix-exa_deps + export HYPRE_INSTALL_DIR=$MY_INSTALL_DIR/hypre + export CSG_INSTALL_DIR=$MY_INSTALL_DIR/csg-deps + export ASCENT_INSTALL_DIR=$MY_INSTALL_DIR/ascent + mkdir -p $HYPRE_INSTALL_DIR + mkdir $CSG_INSTALL_DIR + mkdir $ASCENT_INSTALL_DIR + +#. Load modules and set helpers + + For CPU we'll just build on the head node, but for GPU we connect to an interactive node first: + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + module load cmake/3.23.1 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + module load boost/1.77.0_gnu9.3 + + export CC=$(which mpicc) + export CXX=$(which mpic++) + export F77=$(which mpif77) + export FC=$(which mpifort) + export F90=$(which mpif90) + + .. tab:: GPU + + .. code:: bash + + ## connect to an interactive node + salloc -N 1 -p gpu + + ## load modules + module load cmake/3.23.1 + module load cuda/11.3 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + module load boost/1.77.0_gnu9.3 + + export CC=$(which mpicc) + export CXX=$(which mpic++) + export F77=$(which mpif77) + export FC=$(which mpifort) + export F90=$(which mpif90) + + +#. HYPRE + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + git clone https://github.com/hypre-space/hypre.git + pushd hypre/src/ + git checkout v2.26.0 + ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI + make -j8 install + popd + + .. tab:: GPU + + .. code:: bash + + git clone https://github.com/hypre-space/hypre.git + pushd hypre/src/ + git checkout v2.26.0 + ./configure --prefix=$HYPRE_INSTALL_DIR --without-superlu --disable-bigint --without-openmp --enable-shared --with-MPI --with-cuda --with-gpu-arch='60' --with-cuda-home=$CUDA_HOME --enable-cusparse --enable-curand + make -j32 install + popd + +#. Catch2 + + .. code:: bash + + git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2 + pushd Catch2/ + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR + cd build/ + make -j8 install + popd + +#. GMP + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz + tar -xf gmp-6.2.1.tar.xz + pushd gmp-6.2.1 + ./configure --prefix=$CSG_INSTALL_DIR + make -j8 install + popd + +#. MPFR + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz + tar -xf mpfr-4.1.0.tar.xz + pushd mpfr-4.1.0/ + ./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR + make -j8 install + popd + +#. PEGTL + + .. code:: bash + + git clone --branch 3.2.2 https://github.com/taocpp/PEGTL + pushd PEGTL/ + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR + cd build/ + make -j8 install + popd + +#. Conduit + + .. code:: bash + + git clone --recursive https://github.com/LLNL/conduit.git + pushd conduit/ + git checkout v0.8.4 + mkdir build && cd build + cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DENABLE_OPENMP=OFF -DENABLE_MPI=ON -DENABLE_CUDA=OFF + make -j8 install + popd + +#. Vtk-m + + .. code:: bash + + git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git + pushd vtk-m/ + git checkout v1.9.0 + mkdir build && cd build/ + cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DVTKm_ENABLE_OPENMP=OFF -DVTKm_ENABLE_MPI=ON -DVTKm_ENABLE_CUDA=OFF -DVTKm_USE_64BIT_IDS=OFF -DVTKm_USE_DOUBLE_PRECISION=ON -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON -DVTKm_NO_DEPRECATED_VIRTUAL=ON + make -j8 install + popd + +#. Ascent + + .. code:: bash + + git clone --recursive https://github.com/Alpine-DAV/ascent.git + pushd ascent + mkdir build && cd build/ + cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DCONDUIT_DIR=$ASCENT_INSTALL_DIR -DVTKM_DIR=$ASCENT_INSTALL_DIR -DENABLE_VTKH=ON -DENABLE_FORTRAN=OFF -DENABLE_PYTHON=OFF -DENABLE_DOCS=OFF -DBUILD_SHARED_LIBS=ON + make -j8 install + popd + + +Build mfix +<<<<<<<<<<< + +#. Once the above dependencies are built, clone and load the environment necessary for MFIX-Exa + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + ## Clone + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + mkdir build && cd build/ + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + module load boost/1.77.0_gnu9.3 + + export MY_INSTALL_DIR=$HOME/packages/mfix-exa_deps + export HYPRE_DIR=$MY_INSTALL_DIR/hypre + export HYPRE_ROOT=$HYPRE_DIR + export HYPRE_LIBRARIES=$HYPRE_DIR/lib + export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include + + export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4/include" + + export ASCENT_DIR=$MY_INSTALL_DIR/ascent + export CONDUIT_DIR=$ASCENT_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit + + export CSG_DIR=$MY_INSTALL_DIR/csg-deps + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR + + .. tab:: GPU + + .. code:: bash + + ## Clone + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + mkdir build && cd build/ + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load cuda/11.3 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + + export MY_INSTALL_DIR=$HOME/mfix-exa_deps_gnu9.3_cuda11.3 + export HYPRE_DIR=$MY_INSTALL_DIR/hypre + export HYPRE_ROOT=$HYPRE_DIR + export HYPRE_LIBRARIES=$HYPRE_DIR/lib + export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include + + export ASCENT_DIR=$MY_INSTALL_DIR/ascent + export CONDUIT_DIR=$ASCENT_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit + + export CSG_DIR=$MY_INSTALL_DIR/csg-deps + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR + + export BOOST_ROOT="/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4" + + +#. Build MFIX-Exa + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + cmake -DCMAKE_C_COMPILER=gcc \ + -DCMAKE_CXX_COMPILER=g++ \ + -DCMAKE_Fortran_COMPILER=gfortran \ + -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_GPU_BACKEND=NONE \ + -DMFIX_CSG=yes \ + -DMFIX_HYPRE=yes \ + -DAMReX_ASCENT=yes \ + -DAMReX_CONDUIT=yes \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../mfix + make -j 8 + + ## uncomment to build the pic2dem restarter app + #cmake -j8 --build . --target pic2dem + + .. tab:: GPU + + .. code:: bash + + cmake -DCMAKE_C_COMPILER=gcc \ + -DCMAKE_CXX_COMPILER=g++ \ + -DCMAKE_Fortran_COMPILER=gfortran \ + -DBoost_INCLUDE_DIR="$BOOST_ROOT/include" \ + -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_GPU_BACKEND=CUDA \ + -DAMReX_CUDA_ARCH=6.0 \ + -DGPUS_PER_SOCKET=1 \ + -DGPUS_PER_NODE=2 \ + -DMFIX_CSG=yes \ + -DMFIX_HYPRE=yes \ + -DAMReX_ASCENT=yes \ + -DAMReX_CONDUIT=yes \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + ../mfix + make -j 32 + + + +Building MFIX-Exa: gmake +------------------------- + +Standalone Build +>>>>>>>>>>>>>>>>> + +#. Clone + + For the basic, standalone CPU and GPU builds, first clone the code, + checkout the desired branch and update the submodules + + .. code:: bash + + ## Clone + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + +#. Build mfix + + For a CPU load the necessary modules and build from the head node. For GPU first connect to an interactive node and then build: + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + ## build mfix + make -C exec -j8 COMP=gnu USE_MPI=TRUE USE_OMP=FALSE USE_CUDA=FALSE USE_TINY_PROFILE=FALSE USE_CSG=FALSE USE_HYPRE=FALSE DEBUG=FALSE + + + .. tab:: GPU + + .. code:: bash + + ## connect to an interactive node + salloc -N 1 -p gpu + + ## load Modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load cuda/11.3 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + ## build mfix + make -C exec -j8 COMP=gnu USE_MPI=TRUE USE_OMP=FALSE USE_CUDA=TRUE CUDA_ARCH=6.0 USE_TINY_PROFILE=FALSE USE_CSG=FALSE USE_HYPRE=FALSE DEBUG=FALSE + + +Full Build +>>>>>>>>>>> + +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. + +#. First `Install dependencies`_ + +#. Then, install `libcsgeb` to `$HOME/install_csg_eb` using either cmake or gmake: + + .. tabs:: + + .. tab:: cmake + + .. code:: bash + + ## Clone + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + cd subprojects/csg-eb + + ## Install library + export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4/include" + export CSG_DIR=$MY_INSTALL_DIR/csg-deps + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR` + + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$HOME/install_csg_eb + cd build + make -j8 install + + .. tab:: gmake + + .. code:: bash + + ## Clone + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + cd mfix + git checkout develop + git submodule update --init + + ## Install library + make -C subprojects/csg-eb install DESTDIR=$HOME/install_csg_eb \ + BOOST_HOME=/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4 \ + PEGTL_HOME=$HOME/install \ + CGAL_HOME=$HOME/install \ + CATCH2_HOME=$HOME/install \ + ENABLE_CGAL=TRUE + + +#. Load the necessary modules and environment + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + export MY_INSTALL_DIR=$HOME/packages/mfix-exa_deps + export HYPRE_DIR=$MY_INSTALL_DIR/hypre + export HYPRE_HOME=$HYPRE_DIR + + export ASCENT_DIR=$MY_INSTALL_DIR/ascent + export CONDUIT_DIR=$ASCENT_DIR + + export CSGEB_HOME=$HOME/install_csg_eb + + .. tab:: GPU + + .. code:: bash + + ## load the necessary modules + module purge + module load vgl + module load grace + module load cmake/3.23.1 + module load cuda/11.3 + module load gnu/9.3.0 + module load openmpi/4.0.4_gnu9.3 + + + export MY_INSTALL_DIR=$HOME/mfix-exa_deps + export HYPRE_DIR=$MY_INSTALL_DIR/hypre + export HYPRE_HOME=$HYPRE_DIR + + export ASCENT_DIR=$MY_INSTALL_DIR/ascent + export CONDUIT_DIR=$ASCENT_DIR + + export CSGEB_HOME=$HOME/install_csg_eb + + +#. Build MFIX-Exa + + .. tabs:: + + .. tab:: CPU + + .. code:: bash + + make -C exec -j8 COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=FALSE \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=TRUE \ + USE_HYPRE=TRUE \ + USE_ASCENT=TRUE \ + USE_CONDUIT=TRUE \ + DEBUG=FALSE + + .. tab:: GPU + + .. code:: bash + + make -C exec -j8 COMP=gnu \ + USE_MPI=TRUE \ + USE_OMP=FALSE \ + USE_CUDA=TRUE \ + CUDA_ARCH=6.0 \ + USE_TINY_PROFILE=FALSE \ + USE_CSG=TRUE \ + USE_HYPRE=TRUE \ + USE_ASCENT=TRUE \ + USE_CONDUIT=TRUE \ + DEBUG=FALSE diff --git a/docs/source_docs/getting_started/hpc/Perlmutter.rst b/docs/source_docs/getting_started/hpc/Perlmutter.rst new file mode 100644 index 0000000..aa1c596 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Perlmutter.rst @@ -0,0 +1,4 @@ +Perlmutter +=========== + +Coming soon! diff --git a/docs/source_docs/getting_started/hpc/Polaris.rst b/docs/source_docs/getting_started/hpc/Polaris.rst new file mode 100644 index 0000000..c769b15 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Polaris.rst @@ -0,0 +1,4 @@ +Polaris +======== + +Coming soon! diff --git a/docs/source_docs/getting_started/hpc/Spock.rst b/docs/source_docs/getting_started/hpc/Spock.rst new file mode 100644 index 0000000..c00e700 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Spock.rst @@ -0,0 +1,4 @@ +Spock +====== + +Coming soon! diff --git a/docs/source_docs/getting_started/hpc/Summit.rst b/docs/source_docs/getting_started/hpc/Summit.rst new file mode 100644 index 0000000..8ffe1f2 --- /dev/null +++ b/docs/source_docs/getting_started/hpc/Summit.rst @@ -0,0 +1,4 @@ +Summit +======= + +Coming soon! -- GitLab