diff --git a/docs/source_docs/references/hpc.rst b/docs/source_docs/references/hpc.rst index 2fc47c7a4f3b32752514e84460b8f71be79d8704..0e83eb6e1469277f6aae484c3144ab265f78105c 100644 --- a/docs/source_docs/references/hpc.rst +++ b/docs/source_docs/references/hpc.rst @@ -29,5 +29,4 @@ Follow the guide here instead of the generic instructions. :caption: NETL Systems :maxdepth: 0 - hpc/Joule2 - + hpc/Joule3 diff --git a/docs/source_docs/references/hpc/Crusher.rst b/docs/source_docs/references/hpc/Crusher.rst index 3cf0dde54653e2d48ef7cedbe89aa66548c5292d..5c553bbe01ae8b4228dd7ee2fb06d6b3b9d3d48e 100644 --- a/docs/source_docs/references/hpc/Crusher.rst +++ b/docs/source_docs/references/hpc/Crusher.rst @@ -1,20 +1,20 @@ OLCF Crusher TDS ================ -If this is your first time building MFIX-Exa on Joule2, please +If this is your first time building MFIX-Exa on Joule3, please review the general notes below and `Basics`_ section first. * To access Summit, you will need an `OLCF account `_ - and an RSA token. + and an RSA token. * The name of the MFIX-Exa account is ``cfd122`` -* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The cmake instructions compile to a ``build`` directory. - The gmake instructions compile to a ``exec`` directory. -* For the dependencies, it is assumed that you have set the +* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The cmake instructions compile to a ``build`` directory. + The gmake instructions compile to a ``exec`` directory. +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -24,12 +24,12 @@ review the general notes below and `Basics`_ section first. export ASCENT_INSTALL_DIR=$HOME/ export BOOST_INSTALL_DIR=$HOME/ - to a path that you have read/write access to. - You will need to recall these paths later if you want to build - MFIX-Exa with the optional dependencies. Boost is only needed for - ``PrgEnv-cray``; there is a module available for ``PrgEnv-gnu``. -* After building the ``mfix`` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command + to a path that you have read/write access to. + You will need to recall these paths later if you want to build + MFIX-Exa with the optional dependencies. Boost is only needed for + ``PrgEnv-cray``; there is a module available for ``PrgEnv-gnu``. +* After building the ``mfix`` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the ``build`` directory .. code:: bash @@ -39,9 +39,9 @@ review the general notes below and `Basics`_ section first. .. warning:: - MFIX-Exa currently experiences a floating point exception - when running with ascent support. Make sure all of the - fpe traps in your inputs are off. + MFIX-Exa currently experiences a floating point exception + when running with ascent support. Make sure all of the + fpe traps in your inputs are off. Basics @@ -49,27 +49,27 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -OLCF provides three programming environments on Crusher: -``PrgEnv-amd``, ``PrgEnv-cray``, and ``PrgEnv-gnu``. -We've had the most success with the latter two and only provide +OLCF provides three programming environments on Crusher: +``PrgEnv-amd``, ``PrgEnv-cray``, and ``PrgEnv-gnu``. +We've had the most success with the latter two and only provide build instructions for them below. Note that in the ``PrgEnv-gnu`` -env you will set different compilers if you are compiling for -CPU only (e.g., most of the dependencies) or if you are compiling -for the GPU with HIP-support. +env you will set different compilers if you are compiling for +CPU only (e.g., most of the dependencies) or if you are compiling +for the GPU with HIP-support. -.. tabs:: +.. tabs:: - .. tab:: PrgEnv-cray + .. tab:: PrgEnv-cray - .. code:: bash + .. code:: bash module purge module reset @@ -79,18 +79,18 @@ for the GPU with HIP-support. module load rocm/5.2.0 module load cray-mpich/8.1.21 module load cce/15.0.0 # must reload after rocm - + export MPICH_GPU_SUPPORT_ENABLED=1 - + export CFLAGS="-I${ROCM_PATH}/include" export CXXFLAGS="-I${ROCM_PATH}/include" export LDFLAGS="-L${ROCM_PATH}/lib -lamdhip64" .. tab:: PrgEnv-gnu - Modules + Modules - .. code:: bash + .. code:: bash module purge module reset @@ -104,7 +104,7 @@ for the GPU with HIP-support. CPU compilers - .. code:: bash + .. code:: bash export CC=$(which cc) export CXX=$(which CC) @@ -112,13 +112,13 @@ for the GPU with HIP-support. GPU compilers - .. code:: bash + .. code:: bash export CC=$(which hipcc) export CXX=$(which hipcc) - + export MPICH_GPU_SUPPORT_ENABLED=1 # GPU-aware MPI - + export CFLAGS="-I${MPICH_DIR}/include -I${ROCM_PATH}/include" export CXXFLAGS="-I$OLCF_BOOST_ROOT/include -I${MPICH_DIR}/include -I${ROCM_PATH}/include" export LDFLAGS="-L${MPICH_DIR}/lib -L${CRAY_MPICH_ROOTDIR}/gtl/lib -I${ROCM_PATH}/lib -lmpi -lmpi_gtl_hsa" @@ -128,16 +128,16 @@ for the GPU with HIP-support. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. These instructions are currently -only provided for ``cmake``. +Instructions on building the necessary dependencies are below +and should be successfully installed first. These instructions are currently +only provided for ``cmake``. .. tabs:: - + .. tab:: CPU .. code:: bash @@ -184,7 +184,7 @@ only provided for ``cmake``. export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - + export BOOST_HOME=$MY_INSTALL_DIR/boost #PrgEnv-cray ONLY export Boost_INCLUDE_DIR="-I$BOOST_HOME/include" #PrgEnv-cray ONLY @@ -203,23 +203,23 @@ only provided for ``cmake``. .. tab:: GPU-full .. code:: bash - + export HYPRE_DIR=$HYPRE_INSTALL_DIR export HYPRE_ROOT=$HYPRE_DIR export HYPRE_LIBRARIES=$HYPRE_DIR/lib export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include - + export ASCENT_DIR=$ASCENT_INSTALL_DIR export CONDUIT_DIR=$ASCENT_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit - + export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - + export BOOST_HOME=$MY_INSTALL_DIR/boost #PrgEnv-cray ONLY export Boost_INCLUDE_DIR="-I$BOOST_HOME/include" #PrgEnv-cray ONLY - + export AMREX_AMD_ARCH=gfx90a cmake -DMFIX_MPI=yes \ @@ -239,8 +239,8 @@ only provided for ``cmake``. Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. HYPRE @@ -254,7 +254,7 @@ prior to following any of the full build instructions above. pushd hypre/src/ git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install + make -j8 install popd .. tab:: GPU @@ -278,7 +278,7 @@ prior to following any of the full build instructions above. --with-MPI-lib-dirs="${MPICH_DIR}/lib ${CRAY_MPICH_ROOTDIR}/gtl/lib ${ROCM_PATH}/lib" \ --with-MPI-libs="mpi mpi_gtl_hsa amdhip64" \ --with-MPI-include="${MPICH_DIR}/include {ROCM_PATH}/include" - make -j8 install + make -j8 install popd @@ -343,14 +343,14 @@ prior to following any of the full build instructions above. make -j8 install popd - .. warning:: + .. warning:: - For some reason, something in PEGTL does not set the experimental + For some reason, something in PEGTL does not set the experimental filesystem when installing in ``PrgEnv-gnu`` (but it does in ``PrgEnv-cray``). So you you have to force that manually. One hack to do so is shown below. Make sure that "experimental" precedes filesystem on or near lines 47 and 51 in - ``CSG_INSTALL_DIR/include/tao/pegtl/internal/filesystem.hpp`` + ``CSG_INSTALL_DIR/include/tao/pegtl/internal/filesystem.hpp`` .. code:: bash @@ -362,9 +362,9 @@ prior to following any of the full build instructions above. namespace filesystem = ::std::experimental::filesystem; ... -#. Boost +#. Boost - .. code:: bash + .. code:: bash wget https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.gz tar -zxvf boost_1_81_0.tar.gz @@ -373,9 +373,9 @@ prior to following any of the full build instructions above. ./b2 install --prefix=$BOOST_INSTALL_DIR popd - .. warning:: + .. warning:: - Only needed for PrgEnv-cray. This appears to build with the native + Only needed for PrgEnv-cray. This appears to build with the native gcc/g++ compilers, not the cray wrapped cc/CC compilers, but "it works." @@ -457,7 +457,7 @@ Common Slurm commands: * ``scancel JOBID`` kill a job with id JOBID * ``salloc -N 1 -A CFD122_crusher -J build -t 01:00:00`` grab an interactive node for an hour -Example run scripts: +Example run scripts: .. tabs:: @@ -466,7 +466,7 @@ Example run scripts: .. code:: bash #!/bin/bash -l - + #SBATCH -A CFD122_crusher #SBATCH -J mfix-timing #SBATCH -o job_%x-%j.out @@ -475,11 +475,11 @@ Example run scripts: #SBATCH --exclude=crusher[026,027,028,081,126,114,115] #SBATCH -t 00:05:00 #SBATCH -N 2 - + nodes=2 nrs=12 omp=1 - + module purge module reset module load cpe/22.08 @@ -487,7 +487,7 @@ Example run scripts: module load rocm/5.2.0 module load cray-mpich/8.1.21 module load cce/15.0.0 # must be loaded after rocm - + export OMP_NUM_THREADS=$omp export MPICH_GPU_SUPPORT_ENABLED=1 # remove for cpu only #export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching @@ -496,9 +496,9 @@ Example run scripts: export FI_CXI_REQ_BUF_SIZE=12582912 export FI_CXI_REQ_BUF_MIN_POSTED=6 export FI_CXI_DEFAULT_CQ_SIZE=131072 - + srun -N $nodes -n $nrs -c1 --ntasks-per-gpu=1 --gpu-bind=closest ./mfix inputs > screen.txt - + #cpu: srun -N $nodes -n $nrs -c1 ./mfix inputs > screen.txt .. tab:: PrgEnv-gnu @@ -506,7 +506,7 @@ Example run scripts: .. code:: bash #!/bin/bash -l - + #SBATCH -A CFD122_crusher #SBATCH -J mfix-timing #SBATCH -o job_%x-%j.out @@ -515,11 +515,11 @@ Example run scripts: #SBATCH --exclude=crusher[026,027,028,081,126,114,115] #SBATCH -t 00:05:00 #SBATCH -N 2 - + nodes=2 nrs=12 omp=1 - + module purge module reset module load cpe/22.08 @@ -527,7 +527,7 @@ Example run scripts: module load rocm/5.2.0 module load cray-mpich/8.1.21 module load cce/15.0.0 # must be loaded after rocm - + export OMP_NUM_THREADS=$omp export MPICH_GPU_SUPPORT_ENABLED=1 # remove for cpu only #export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching @@ -536,10 +536,7 @@ Example run scripts: export FI_CXI_REQ_BUF_SIZE=12582912 export FI_CXI_REQ_BUF_MIN_POSTED=6 export FI_CXI_DEFAULT_CQ_SIZE=131072 - - srun -N $nodes -n $nrs -c1 --ntasks-per-gpu=1 --gpu-bind=closest ./mfix inputs > screen.txt - - #cpu: srun -N $nodes -n $nrs -c1 ./mfix inputs > screen.txt - + srun -N $nodes -n $nrs -c1 --ntasks-per-gpu=1 --gpu-bind=closest ./mfix inputs > screen.txt + #cpu: srun -N $nodes -n $nrs -c1 ./mfix inputs > screen.txt diff --git a/docs/source_docs/references/hpc/Frontier.rst b/docs/source_docs/references/hpc/Frontier.rst index 7aa0cb14c3addbcb92aa1ed6652d97d27d0ceb80..aaeb62a940481059edcd7e08d5d05e746485c058 100644 --- a/docs/source_docs/references/hpc/Frontier.rst +++ b/docs/source_docs/references/hpc/Frontier.rst @@ -1,20 +1,20 @@ OLCF Frontier ============= -If this is your first time building MFIX-Exa on Joule2, please +If this is your first time building MFIX-Exa on Joule3, please review the general notes below and `Basics`_ section first. * To access Summit, you will need an `OLCF account `_ - and an RSA token. + and an RSA token. * The name of the MFIX-Exa account is ``cfd122`` -* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. - You may have to decrease this value if there is high traffic - or you may want to increase this value if you are on a compute - node interactively. -* The instructions assume that you are compiling in a directory +* These instructions build MFIX-Exa on the login nodes using ``-j8`` CPUs. + You may have to decrease this value if there is high traffic + or you may want to increase this value if you are on a compute + node interactively. +* The instructions assume that you are compiling in a directory inside of the ``mfix`` directory. -* For the dependencies, it is assumed that you have set the +* For the dependencies, it is assumed that you have set the following environment variables: .. code:: bash @@ -23,25 +23,25 @@ review the general notes below and `Basics`_ section first. export CSG_INSTALL_DIR=$HOME/ export ASCENT_INSTALL_DIR=$HOME/ - to a path that you have read/write access to. - You will need to recall these paths later if you want to build + to a path that you have read/write access to. + You will need to recall these paths later if you want to build MFIX-Exa with the optional dependencies. -* After building the ``mfix`` executable (with cmake), you can - build the PIC-to-DEM restarter app by executing the following command +* After building the ``mfix`` executable (with cmake), you can + build the PIC-to-DEM restarter app by executing the following command in the ``build`` directory .. code:: bash cmake --build . --target pic2dem -* These instructions have targeded the Cray Programming Environment. - The GNU PE _should_ also work, but it has not been tested. Please - see previous notes on Crusher TDS. +* These instructions have targeded the Cray Programming Environment. + The GNU PE _should_ also work, but it has not been tested. Please + see previous notes on Crusher TDS. .. warning:: - MFIX-Exa currently experiences a floating point exception - when running with ascent support. Make sure all of the - fpe traps in your inputs are off. + MFIX-Exa currently experiences a floating point exception + when running with ascent support. Make sure all of the + fpe traps in your inputs are off. Basics @@ -49,23 +49,23 @@ Basics Source code ~~~~~~~~~~~ - + Before building, first obtain a copy of the source code -following the instructions on the +following the instructions on the `MFIX-Exa website. `_ Modules ~~~~~~~ -These build notes target the (default) ``PrgEnv-cray`` -programming environment and the ``cmake`` build method. +These build notes target the (default) ``PrgEnv-cray`` +programming environment and the ``cmake`` build method. -.. tabs:: +.. tabs:: - .. tab:: CPU + .. tab:: CPU - .. code:: bash + .. code:: bash module purge module reset @@ -75,10 +75,10 @@ programming environment and the ``cmake`` build method. module load cray-mpich/8.1.23 module load gmp/6.2.1 module load boost/1.79.0 - - .. tab:: GPU - .. code:: bash + .. tab:: GPU + + .. code:: bash module purge module reset @@ -90,9 +90,9 @@ programming environment and the ``cmake`` build method. module load cce/15.0.0 # must reload after rocm module load gmp/6.2.1 module load boost/1.79.0 - + export MPICH_GPU_SUPPORT_ENABLED=1 - + export CFLAGS="-I${ROCM_PATH}/include" export CXXFLAGS="-I${ROCM_PATH}/include" export LDFLAGS="-L${ROCM_PATH}/lib -lamdhip64" @@ -100,16 +100,16 @@ programming environment and the ``cmake`` build method. Building MFIX-Exa ----------------- -The commands below are the superbuild instructions, i.e., -AMReX is built as part of the MFIX-Exa build process. -To build MFIX-Exa with hypre, csg and/or ascent dependencies, +The commands below are the superbuild instructions, i.e., +AMReX is built as part of the MFIX-Exa build process. +To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. -Instructions on building the necessary dependencies are below -and should be successfully installed first. These instructions are currently -only provided for ``cmake``. +Instructions on building the necessary dependencies are below +and should be successfully installed first. These instructions are currently +only provided for ``cmake``. .. tabs:: - + .. tab:: CPU .. code:: bash @@ -156,7 +156,7 @@ only provided for ``cmake``. export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - + export Boost_INCLUDE_DIR="-I$OLCF_BOOST_ROOT/include" cmake -DCMAKE_BUILD_TYPE=Release \ @@ -174,22 +174,22 @@ only provided for ``cmake``. .. tab:: GPU-full .. code:: bash - + export HYPRE_DIR=$HYPRE_INSTALL_DIR export HYPRE_ROOT=$HYPRE_DIR export HYPRE_LIBRARIES=$HYPRE_DIR/lib export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include - + export ASCENT_DIR=$ASCENT_INSTALL_DIR export CONDUIT_DIR=$ASCENT_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit - + export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - + export Boost_INCLUDE_DIR="-I$OLCF_BOOST_ROOT/include" - + export AMREX_AMD_ARCH=gfx90a cmake -DCMAKE_BUILD_TYPE=Release \ @@ -209,8 +209,8 @@ only provided for ``cmake``. Optional build dependencies --------------------------- -The following dependencies need to be built and installed -prior to following any of the full build instructions above. +The following dependencies need to be built and installed +prior to following any of the full build instructions above. #. HYPRE @@ -224,7 +224,7 @@ prior to following any of the full build instructions above. pushd hypre/src/ git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI - make -j8 install + make -j8 install popd .. tab:: GPU @@ -248,7 +248,7 @@ prior to following any of the full build instructions above. --with-MPI-lib-dirs="${MPICH_DIR}/lib ${CRAY_MPICH_ROOTDIR}/gtl/lib ${ROCM_PATH}/lib" \ --with-MPI-libs="mpi mpi_gtl_hsa amdhip64" \ --with-MPI-include="${MPICH_DIR}/include {ROCM_PATH}/include" - make -j8 install + make -j8 install popd @@ -380,19 +380,19 @@ Common Slurm commands: * ``scancel JOBID`` kill a job with id JOBID * ``salloc -N 1 -q debug -A CFD122 -J build -t 01:00:00`` grab an interactive node for an hour -Example run scripts: +Example run scripts: .. warning:: - Even when running a CPU-only MFIX-Exa build, you **do** still have to - load ROCm to avoid a missing loading shared libraries: libamdhip64.so.5) - error. The ``FI_*`` flags seem to be unnecessary for CPU-only runs, - but this has not been extensively tested yet. + Even when running a CPU-only MFIX-Exa build, you **do** still have to + load ROCm to avoid a missing loading shared libraries: libamdhip64.so.5) + error. The ``FI_*`` flags seem to be unnecessary for CPU-only runs, + but this has not been extensively tested yet. .. code:: bash #!/bin/bash -l - + #SBATCH -A CFD122 #SBATCH -J runxyz #SBATCH -o job_%x-%j.out @@ -400,11 +400,11 @@ Example run scripts: #SBATCH --threads-per-core=1 #SBATCH -t 00:15:00 #SBATCH -N 2 - + nodes=2 nrs=16 omp=1 - + module purge module reset module load cmake/3.23.2 @@ -415,7 +415,7 @@ Example run scripts: module load cce/15.0.0 # must be loaded after rocm module load gmp/6.2.1 module load boost/1.79.0 - + export OMP_NUM_THREADS=$omp export MPICH_GPU_SUPPORT_ENABLED=1 # remove for cpu only #export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching @@ -424,8 +424,7 @@ Example run scripts: export FI_CXI_REQ_BUF_SIZE=12582912 export FI_CXI_REQ_BUF_MIN_POSTED=6 export FI_CXI_DEFAULT_CQ_SIZE=131072 - + srun -N $nodes -n $nrs -c1 --ntasks-per-gpu=1 --gpu-bind=closest ./mfix inputs > screen.txt - - #cpu srun -N $nodes -n $nrs -c1 ./mfix inputs > screen.txt + #cpu srun -N $nodes -n $nrs -c1 ./mfix inputs > screen.txt diff --git a/docs/source_docs/references/hpc/Joule2.rst b/docs/source_docs/references/hpc/Joule3.rst similarity index 94% rename from docs/source_docs/references/hpc/Joule2.rst rename to docs/source_docs/references/hpc/Joule3.rst index 56de8c9236c948d376fb7a7b71c3bb295acedb90..fd7c43df0ae8a1c332cf7a37d2c9b09f789c6d65 100644 --- a/docs/source_docs/references/hpc/Joule2.rst +++ b/docs/source_docs/references/hpc/Joule3.rst @@ -1,10 +1,10 @@ Joule 2.0 ========== -If this is your first time building MFIX-Exa on Joule2, please +If this is your first time building MFIX-Exa on Joule3, please review the general notes below and `Basics`_ section first. -* To access Joule2, you will need an NETL HPC account, +* To access Joule3, you will need an NETL HPC account, a yubikey (2FA) and the `HPC client. `_ * These instructions build MFIX-Exa on the login nodes using `-j8` CPUs. @@ -54,15 +54,15 @@ following modules .. code:: bash module purge - module load cmake/3.23.1 - module load gnu/9.3.0 - module load openmpi/4.0.4_gnu9.3 + module load cmake + module load gnu/13.2.0 + module load openmpi/4.1.5/gnu/13.2.0 The GPU-enabled builds additionally require .. code:: bash - module load cuda/11.3 + module load cuda Full builds that utilize external dependencies, also require setting certain environment variables as discussed below. @@ -89,6 +89,8 @@ cmake .. code:: bash + mkdir build; cd build + cmake -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -DMFIX_MPI=yes \ @@ -105,6 +107,8 @@ cmake .. code:: bash + mkdir build; cd build + cmake -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ -DMFIX_MPI=yes \ @@ -138,8 +142,9 @@ cmake export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR - module load boost/1.77.0_gnu9.3 - export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4/include" + export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.83.0/gnu/13.2.0/openmpi/4.1.5/include/" + + mkdir build; cd build cmake -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ @@ -159,6 +164,8 @@ cmake .. code:: bash + mkdir build; cd build + export HYPRE_DIR=$HYPRE_INSTALL_DIR export HYPRE_ROOT=$HYPRE_DIR export HYPRE_LIBRARIES=$HYPRE_DIR/lib @@ -173,7 +180,7 @@ cmake export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR module load boost/1.77.0_gnu9.3 - export BOOST_ROOT="/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4" + export BOOST_ROOT="/nfs/apps/Libraries/Boost/1.83.0/gnu/13.2.0/openmpi/4.1.5" cmake -DCMAKE_C_COMPILER=gcc \ -DCMAKE_CXX_COMPILER=g++ \ @@ -312,7 +319,6 @@ prior to following any of the full build instructions above. git clone https://github.com/hypre-space/hypre.git pushd hypre/src/ - git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI make -j8 install popd @@ -323,7 +329,6 @@ prior to following any of the full build instructions above. git clone https://github.com/hypre-space/hypre.git pushd hypre/src/ - git checkout v2.26.0 ./configure --prefix=$HYPRE_INSTALL_DIR \ --without-superlu \ --disable-bigint \ @@ -342,7 +347,7 @@ prior to following any of the full build instructions above. .. code:: bash - git clone --depth 1 --branch v3.11.0 https://github.com/catchorg/Catch2 + git clone --depth 1 https://github.com/catchorg/Catch2 pushd Catch2/ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR cd build/ @@ -375,7 +380,7 @@ prior to following any of the full build instructions above. .. code:: bash - git clone --depth 1 --branch v6.4 https://github.com/CGAL/cgal + git clone --depth 1 https://github.com/CGAL/cgal pushd cgal/ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR cd build/ @@ -387,7 +392,7 @@ prior to following any of the full build instructions above. .. code:: bash - git clone --branch 3.2.8 https://github.com/taocpp/PEGTL + git clone https://github.com/taocpp/PEGTL pushd PEGTL/ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR cd build/ @@ -409,7 +414,7 @@ prior to following any of the full build instructions above. module load boost/1.77.0_gnu9.3 - export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4/include" + export Boost_INCLUDE_DIR="-I/nfs/apps/Libraries/Boost/1.83.0/gnu/13.2.0/openmpi/4.1.5/include" export CSG_DIR=$CSG_INSTALL_DIR export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR @@ -424,7 +429,7 @@ prior to following any of the full build instructions above. .. code:: bash make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \ - BOOST_HOME=/nfs/apps/Libraries/Boost/1.77.0/gnu/9.3.0/openmpi/4.0.4 \ + BOOST_HOME=/nfs/apps/Libraries/Boost/1.83.0/gnu/13.2.0/openmpi/4.1.4 \ PEGTL_HOME=$CSG_DIR \ CGAL_HOME=$CSG_DIR \ CATCH2_HOME=$CSG_DIR \ @@ -436,7 +441,6 @@ prior to following any of the full build instructions above. git clone --recursive https://github.com/LLNL/conduit.git pushd conduit/ - git checkout v0.8.6 mkdir build && cd build cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ -DENABLE_OPENMP=OFF \ @@ -452,7 +456,6 @@ prior to following any of the full build instructions above. git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git pushd vtk-m/ - git checkout v1.9.0 mkdir build && cd build/ cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ -DVTKm_ENABLE_OPENMP=OFF \ @@ -472,7 +475,6 @@ prior to following any of the full build instructions above. git clone --recursive https://github.com/Alpine-DAV/ascent.git pushd ascent - git checkout v0.9.0 mkdir build && cd build/ cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \ -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \ diff --git a/docs/source_docs/references/units.rst b/docs/source_docs/references/units.rst index 5ece703964824a1be37f0c5e0bcbcf987884eb25..2355ef8ee33ae443a5a93e9ab543c04747ffde1e 100644 --- a/docs/source_docs/references/units.rst +++ b/docs/source_docs/references/units.rst @@ -9,43 +9,46 @@ units. +--------------------------------+-----------------------------------------+ | Physical quantity | MFIX-Exa SI unit | +================================+=========================================+ ++--------------------------------+-----------------------------------------+ | amount of substance | mole [:math:`mol`] | +--------------------------------+-----------------------------------------+ -| length | meter [:math:`m`] | +| density | [:math:`kg \cdot m^{-3}`] | +--------------------------------+-----------------------------------------+ -| mass | kilogram [:math:`kg`] | +| diffusivity | [:math:`m^2 \cdot s^{-1}`] | +--------------------------------+-----------------------------------------+ -| time | second [:math:`s`] | +| dynamic viscosity | [:math:`Pa \cdot s`] | ++--------------------------------+-----------------------------------------+ +| energy | Joule [:math:`J`] | +--------------------------------+-----------------------------------------+ | force | Newton [:math:`N`] | +--------------------------------+-----------------------------------------+ -| temperature | Kelvin [:math:`K`] | +| granular temperature | [:math m^{2} \cdot s{-2} | +--------------------------------+-----------------------------------------+ -| pressure | Pascal [:math:`Pa`] | +| heat of formation | [:math:`J \cdot kg^{-1}`] | +--------------------------------+-----------------------------------------+ -| energy | Joule [:math:`J`] | +| length | meter [:math:`m`] | +--------------------------------+-----------------------------------------+ -| power | Watt [:math:`W`] | +| mass | kilogram [:math:`kg`] | +--------------------------------+-----------------------------------------+ -| density | [:math:`kg \cdot m^{-3}`] | +| molecular weight | [:math:`kg \cdot mol^{-1}`] | +--------------------------------+-----------------------------------------+ -| velocity | [:math:`m \cdot s^{-1}`] | +| power | Watt [:math:`W`] | +--------------------------------+-----------------------------------------+ -| dynamic viscosity | [:math:`Pa \cdot s`] | +| pressure | Pascal [:math:`Pa`] | +--------------------------------+-----------------------------------------+ -| diffusivity | [:math:`m^2 \cdot s^{-1}`] | +| reaction rate (Euler-Euler) | [:math:`mol \cdot m^{-3} \cdot s^{-1}`] | +--------------------------------+-----------------------------------------+ -| specific heat capacity | [:math:`J \cdot kg^{-1} \cdot K^{-1}`] | +| reaction rate (Euler-Lagrange) | [:math:`mol \cdot s^{-1}`] | +--------------------------------+-----------------------------------------+ -| thermal conductivity | [:math:`W \cdot m^{-1} \cdot K^{-1}`] | +| specific heat capacity | [:math:`J \cdot kg^{-1} \cdot K^{-1}`] | +--------------------------------+-----------------------------------------+ | spring coefficient | [:math:`N \cdot m^{-1}`] | +--------------------------------+-----------------------------------------+ -| molecular weight | [:math:`kg \cdot mol^{-1}`] | +| temperature | Kelvin [:math:`K`] | +--------------------------------+-----------------------------------------+ -| heat of formation | [:math:`J \cdot kg^{-1}`] | +| thermal conductivity | [:math:`W \cdot m^{-1} \cdot K^{-1}`] | +--------------------------------+-----------------------------------------+ -| reaction rate (Euler-Euler) | [:math:`mol \cdot m^{-3} \cdot s^{-1}`] | +| time | second [:math:`s`] | +--------------------------------+-----------------------------------------+ -| reaction rate (Euler-Lagrange) | [:math:`mol \cdot s^{-1}`] | +| velocity | [:math:`m \cdot s^{-1}`] | +--------------------------------+-----------------------------------------+ diff --git a/docs/source_docs/user_guide/inputs/initial_conditions.rst b/docs/source_docs/user_guide/inputs/initial_conditions.rst index a28137db1be5380a5593990e9bd23122f442a0c5..7c94000bf717e77ecf97406fcc4255cf3dca19f4 100644 --- a/docs/source_docs/user_guide/inputs/initial_conditions.rst +++ b/docs/source_docs/user_guide/inputs/initial_conditions.rst @@ -69,6 +69,9 @@ using the compound prefix ``ic.[region_name]``: | | (NOTE: ``oneper`` is equivalent to ``1-cube`` and ``eightper`` | | | | | to ``2-cube``) | | | +----------------------+------------------------------------------------------------------+-------------+-----------+ +| granular_temperature | Adds specified granular temperature to particles in the IC | Real | | +| | region. | | | ++----------------------+------------------------------------------------------------------+-------------+-----------+ For each initial condition region, the solid inputs are defined using the compound prefix ``ic.[region_name].[solid_name]``: