diff --git a/docs/source_docs/references/hpc.rst b/docs/source_docs/references/hpc.rst index 2fc47c7a4f3b32752514e84460b8f71be79d8704..dc52b4d8032608b07605bd33f1f0bfa93ecdb73f 100644 --- a/docs/source_docs/references/hpc.rst +++ b/docs/source_docs/references/hpc.rst @@ -30,4 +30,5 @@ Follow the guide here instead of the generic instructions. :maxdepth: 0 hpc/Joule2 + hpc/Joule3 diff --git a/docs/source_docs/references/hpc/Joule3.rst b/docs/source_docs/references/hpc/Joule3.rst new file mode 100644 index 0000000000000000000000000000000000000000..2a9d350ec9c4fa43a078a7dcd4e468052a9daca6 --- /dev/null +++ b/docs/source_docs/references/hpc/Joule3.rst @@ -0,0 +1,394 @@ +Joule 3.0 +========== + +Congratulations on getting the access to Joule3.0. You have accomplished +the first step to use the platform for exascale computing. +Follow these notes to build MFIX-Exa code: + +Source code +----------- + +Request access and download the source code following the instructions on the +`MFIX-Exa website. `_ + +Essentials +---------- + +The code require some essential modules such as cmake, gnu, openmpi and cuda +which are readily available on Joule3.0. Use the following set of commands to load in your environment: + +.. code:: bash + + module purge + module load cmake/3.29.3 + module load gnu/13.2.0 + module load openmpi/4.1.5/gnu/13.2.0 + module load cuda/12.4 # If you want the code to be GPU compatible + +Important NOTES: +---------------- + +- Installation of all dependencies and code is tested with the version of modules shown above. +- We are using 8 cores using '-j8' for all make commands. User can change the number of cores based on the core availability on the login node. +- The following instructions of installation are not test for GPU compatibility, and would require modifications. + +Dependencies +------------ + +Dependencies are only required to be installed once on Joule3.0. If already installed, skip to the next section. + +You need to set certain environment variables before building the dependencies. + +.. code:: bash + + export CC=$(which mpicc) + export CXX=$(which mpic++) + export F77=$(which mpif77) + export FC=$(which mpifort) + export F90=$(which mpif90) + +You should create a 'dep_installed' directory for the compiled packages and a 'dep_source_code' for the source code of each package to keep everything organized, it is optional but recommended. You only need to copy-paste-execute most of the instructions given below, except replacing few path marked as 'CHANGE_HERE' + +.. code:: bash + + export DEP_SOURCE_DIR=$HOME//dep_source_code # CHANGE_HERE + export DEP_INSTALL_DIR=$HOME//dep_installed # CHANGE_HERE + export MFIX_SOURCE_DIR=$HOME/ # CHANGE_HERE + export MFIX_BUILD_DIR=$HOME/ # CHANGE_HERE + + cd $DEP_SOURCE_DIR + +#. HYPRE + + .. code:: bash + + git clone https://github.com/hypre-space/hypre.git + pushd hypre/src/ + git checkout v2.26.0 + + export HYPRE_INSTALL_DIR=$DEP_INSTALL_DIR/HYPRE/2_26_0 + + ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI + make -j8 install + + popd + +#. CATCH2 + + .. code:: bash + + git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2 + pushd Catch2/ + + export CATCH2_INSTALL_DIR=$DEP_INSTALL_DIR/Catch2/2_13_7 + + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CATCH2_INSTALL_DIR + cd build + make -j8 install + + popd + + +#. GMP + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz + tar -xf gmp-6.2.1.tar.xz + pushd gmp-6.2.1 + + export GMP_INSTALL_DIR=$DEP_INSTALL_DIR/gmp/6_2_1 + + ./configure --prefix=$GMP_INSTALL_DIR + make -j8 install + + popd + + +#. MPFR + + .. code:: bash + + wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz + tar -xf mpfr-4.1.0.tar.xz + pushd mpfr-4.1.0/ + + export MPFR_INSTALL_DIR=$DEP_INSTALL_DIR/mpfr/4_1_0 + + ./configure --with-gmp=$GMP_INSTALL_DIR --prefix=$MPFR_INSTALL_DIR + make -j8 install + + popd + +#. CGAL + + .. code:: bash + + git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal + pushd cgal/ + + export CGAL_INSTALL_DIR=$DEP_INSTALL_DIR/cgal/5_3_0 + + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CGAL_INSTALL_DIR + cd build/ + make -j8 install + + popd + +#. PEGTL + + .. code:: bash + + git clone --branch 3.2.2 https://github.com/taocpp/pegtl + pushd pegtl/ + + export PEGTL_INSTALL_DIR=$DEP_INSTALL_DIR/pegtl/3_2_2 + + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$PEGTL_INSTALL_DIR + cd build/ + make -j8 install + + popd + +#. Boost + + .. code:: bash + + wget --no-check-certificate https://archives.boost.io/release/1.85.0/source/boost_1_85_0.tar.gz + tar -xzvf boost_1_85_0.tar.gz + pushd boost_1_85_0/ + + export BOOST_INSTALL_DIR=$DEP_INSTALL_DIR/boost/1_85_0 + + ./bootstrap.sh --prefix=$BOOST_INSTALL_DIR + ./b2 install + + popd + + +#. CSG EB + + .. code:: bash + + git clone https://mfix.netl.doe.gov/gitlab/exa/csg-eb + pushd csg-eb + + CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CATCH2_INSTALL_DIR + CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$GMP_INSTALL_DIR + CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$MPFR_INSTALL_DIR + CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CGAL_INSTALL_DIR + CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$PEGTL_INSTALL_DIR/share/pegtl/cmake + export CMAKE_PREFIX_PATH + export CSGEB_INSTALL_DIR=$DEP_INSTALL_DIR/csgeb + + cmake -Bbuild-csg-eb-release \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=$CSGEB_INSTALL_DIR \ + -DBoost_INCLUDE_DIR=$BOOST_INSTALL_DIR/include + cmake --build build-csg-eb-release --target install + + popd + +#. Conduit + + .. code:: bash + + git clone --recursive https://github.com/LLNL/conduit.git + pushd conduit + git checkout v0.8.6 + + export CONDUIT_INSTALL_DIR=$DEP_INSTALL_DIR/conduit_0_8_6 + + cmake -S ./src -B build -DCMAKE_INSTALL_PREFIX=$CONDUIT_INSTALL_DIR \ + -DENABLE_OPENMP=OFF \ + -DENABLE_MPI=ON \ + -DENABLE_CUDA=OFF \ + -DCMAKE_BUILD_TYPE=Release + + cd build + make -j8 install + + popd + + +#. VTK-m + + .. code:: bash + + git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git + pushd vtk-m + git checkout v1.9.0 + + export VTK_INSTALL_DIR=$DEP_INSTALL_DIR/vtk-m_1_9_0 + + cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$VTK_INSTALL_DIR \ + -DVTKm_ENABLE_OPENMP=OFF \ + -DVTKm_ENABLE_MPI=ON \ + -DVTKm_ENABLE_CUDA=OFF \ + -DVTKm_USE_64BIT_IDS=OFF \ + -DVTKm_USE_DOUBLE_PRECISION=ON \ + -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \ + -DVTKm_NO_DEPRECATED_VIRTUAL=ON \ + -DCMAKE_BUILD_TYPE=Release + cd build + make -j8 install + + popd + + +#. Ascent + + .. code:: bash + + git clone --recursive https://github.com/Alpine-DAV/ascent.git + pushd ascent + git checkout v0.9.0 + + export ASCENT_INSTALL_DIR=$DEP_INSTALL_DIR/ascent_0_9_0 + + cmake -S ./src -B build -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ + -DCONDUIT_DIR=$CONDUIT_INSTALL_DIR \ + -DVTKM_DIR=$VTK_INSTALL_DIR \ + -DENABLE_VTKH=ON \ + -DENABLE_FORTRAN=OFF \ + -DENABLE_PYTHON=OFF \ + -DENABLE_DOCS=OFF \ + -DBUILD_SHARED_LIBS=ON \ + -DCMAKE_BUILD_TYPE=Release \ + -DENABLE_GTEST=OFF \ + -DENABLE_TESTS=OFF + cd build + make -j8 install + + popd + +If you want to change the version of XYZABC dependency in future to test the exa code, follow the same installation process and only change the version in the variable 'XYZABC_INSTALL_DIR' to ensure you do not break your previous installation, so you can always go back. + + +You have successfully installed all the libraries required for the installation of MFIX-Exa code. + +Building MFIX-Exa +----------------- + +We build the MFIX-Exa in superbuild mode, i.e., AMReX is build as part of the MFIX-Exa build process. + +.. code:: bash + + export HYPRE_DIR=$HYPRE_INSTALL_DIR + export HYPRE_ROOT=$HYPRE_DIR + export HYPRE_LIBRARIES=$HYPRE_DIR/lib + export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include + + export ASCENT_DIR=$ASCENT_INSTALL_DIR + export CONDUIT_DIR=$CONDUIT_INSTALL_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CONDUIT_DIR/lib/cmake/conduit + + export CSG_DIR=$CSG_INSTALL_DIR + export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR + + export Boost_INCLUDE_DIR=$BOOST_INSTALL_DIR/include + + cd $MFIX_SOURCE_DIR + + git clone https://mfix.netl.doe.gov/gitlab/exa/mfix.git + + cd mfix/src + + cmake -B $MFIX_BUILD_DIR -DCMAKE_C_COMPILER=gcc \ + -DCMAKE_CXX_COMPILER=g++ \ + -DMFIX_MPI=yes \ + -DMFIX_OMP=no \ + -DMFIX_GPU_BACKEND=NONE \ + -DAMReX_TINY_PROFILE=no \ + -DCMAKE_BUILD_TYPE=Release \ + -DBoost_INCLUDE_DIR=$BOOST_INSTALL_DIR/include \ + -DMFIX_CSG=yes \ + -DMFIX_HYPRE=yes \ + -DAMReX_ASCENT=yes \ + -DAMReX_CONDUIT=yes \ + ../ + + cd $MFIX_BUILD_DIR + + make -j8 + +The MFIX-Exa build is now complete! + +The build creates an executable (mfix.exe) in the '$MFIX_BUILD_DIR' that you can use to run MFIX-Exa cases. (OPTIONAL) You can create an alias in ~/.bashrc as shown below to access MFIX-Exa executable from any path. + +.. code:: bash + + echo "alias mfixexa='$MFIX_BUILD_DIR/mfix'" > ~/.bashrc + +Running Jobs +------------ + +Common Slurm commands: + +* ``sinfo`` see available/allocated resources +* ``sbatch runit_cpu.sh`` submit a cpu job to the queue +* ``squeue -u USER`` check job status of user USER +* ``squeue -p PARTITION`` check job status of partition PARTITION +* ``scancel JOBID`` kill a job with id JOBID +* ``salloc -N 1 -p gpu`` grab a GPU node interactively (for up to 48 hrs) +* ``salloc -N 2 -p dev -q dev`` grab two development nodes (for up to 2 hrs) + +Example run scripts: + +.. tabs:: + + .. tab:: CPU + + .. code:: bash + + #!/bin/bash -l + + ##Accounting + #SBATCH --partition=general #bigmem, dev + #SBATCH --qos=normal #long, dev + + ##Submission + #SBATCH --nodes=1 + #SBATCH --job-name="mfix-exa-run" + #SBATCH --output=job.out + #SBATCH --mail-user=first.last@netl.doe.gov + #SBATCH --mail-type=ALL + + ##Load Modules + module purge + module load gnu/13.2.0 + module load openmpi/4.1.5/gnu/13.2.0 + + ##Run the program + mpirun -np 36 ./mfix inputs > screen.txt + + + .. tab:: GPU + + .. code:: bash + + #!/bin/bash -l + + ##Accounting + #SBATCH --partition=gpu + #SBATCH --qos=normal #long + + ##Submission + #SBATCH --nodes=2 + #SBATCH --ntasks-per-node=2 + #SBATCH --ntasks-per-socket=1 + #SBATCH --job-name="mfix-exa-run" + #SBATCH --output=job.out + #SBATCH --mail-user=first.last@netl.doe.gov + #SBATCH --mail-type=ALL + + ##Load Modules + module purge + module load cuda/12.4 + module load gnu/13.2.0 + module load openmpi/4.1.5/gnu/13.2.0 + + ##Run the program + mpirun -np 4 -npersocket 1 ./mfix inputs > screen.txt +