PSC Bridges-2

If this is your first time building MFIX-Exa on Bridges-2, please review the general notes below and Basics section first.

To access Delta-2, you will need PSC account which can be administered by the NSF ACCESS allocation management system.
You can find the name of your account(s) and the current balance with the accounts command.
These instructions build MFIX-Exa on the login nodes using -j8 CPUs. You may have to decrease this value if there is high traffic or you may want to increase this value if you are on a compute node interactively.

For the dependencies, it is assumed that you have set the following environment variables:

export HYPRE_INSTALL_DIR=$HOME/<path/to/my/hypre-install-dir>
export CSG_INSTALL_DIR=$HOME/<path/to/my/csg-dep-install-dir>
export CSG_LIB_DIR=$HOME/<path/to/my/csg-lib-install-dir>
export ASCENT_INSTALL_DIR=$HOME/<path/to/my/ascent-install-dir>

to a path that you have read/write access to. You will need to recall these paths later if you want to build MFIX-Exa with the optional dependencies.

Basics

Source code

Before building, first obtain a copy of the source code following the instructions on the MFIX-Exa website.

Modules

There are (basically) no modules loaded by default on Bridges-2.

module load gcc/10.2.0                 # PrgEnv
module load openmpi/4.0.5-gcc10.2.0    # MPI
module load cuda/11.7.1                # for GPU
module load boost/1.75-gcc10.2.0       # for CSG

Building MFIX-Exa

CMake 3.20 or higher is required. You are running version 3.11.4

The commands below are the superbuild instructions, i.e., AMReX is built as part of the MFIX-Exa build process. To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. Instructions on building the necessary dependencies are below and should be successfully installed first. Only the gmake method of building MFIX-Exa is currently recommended.

cp -r exec exec.cpu
make -C exec.cpu -j8 \
     COMP=gnu \
     USE_MPI=TRUE \
     USE_OMP=FALSE \
     USE_CUDA=FALSE \
     USE_TINY_PROFILE=FALSE \
     USE_CSG=FALSE \
     USE_HYPRE=FALSE \
     DEBUG=FALSE

cp -r exec exec.gpu
make -C exec.gpu -j8 \
     COMP=gnu \
     USE_MPI=TRUE \
     USE_OMP=FALSE \
     USE_CUDA=TRUE \
     CUDA_ARCH=70 \
     USE_TINY_PROFILE=FALSE \
     USE_CSG=FALSE \
     USE_HYPRE=FALSE \
     DEBUG=FALSE

export HYPRE_DIR=$HYPRE_INSTALL_DIR
export HYPRE_HOME=$HYPRE_DIR

export ASCENT_DIR=$ASCENT_INSTALL_DIR
export CONDUIT_DIR=$ASCENT_DIR

export CSGEB_HOME=$CSG_LIB_DIR
export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib"
cp -r exec exec.cpu.full
make -C exec.cpu.full -j8 \
     COMP=gnu \
     USE_MPI=TRUE \
     USE_OMP=FALSE \
     USE_CUDA=FALSE \
     USE_TINY_PROFILE=FALSE \
     USE_CSG=TRUE \
     USE_HYPRE=TRUE \
     USE_ASCENT=TRUE \
     USE_CONDUIT=TRUE \
     DEBUG=FALSE

export HYPRE_DIR=$HYPRE_INSTALL_DIR
export HYPRE_HOME=$HYPRE_DIR

export ASCENT_DIR=$ASCENT_INSTALL_DIR
export CONDUIT_DIR=$ASCENT_DIR

export CSGEB_HOME=$CSG_LIB_DIR
export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib"
cp -r exec exec.gpu.full
make -C exec.gpu.full -j8 COMP=gnu \
     USE_MPI=TRUE \
     USE_OMP=FALSE \
     USE_CUDA=TRUE \
     CUDA_ARCH=70 \
     USE_TINY_PROFILE=FALSE \
     USE_CSG=TRUE \
     USE_HYPRE=TRUE \
     USE_ASCENT=TRUE \
     USE_CONDUIT=TRUE \
     DEBUG=FALSE

Optional build dependencies

The following dependencies need to be built and installed prior to following any of the full build instructions above.

HYPRE

git clone https://github.com/hypre-space/hypre.git
pushd hypre/src/
git checkout v2.26.0
./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI
make -j8 install
popd

git clone https://github.com/hypre-space/hypre.git
pushd hypre/src/
git checkout v2.26.0
./configure --prefix=$HYPRE_INSTALL_DIR \
            --without-superlu \
            --disable-bigint \
            --without-openmp \
            --enable-unified-memory \
            --with-MPI \
            --with-cuda \
            --with-gpu-arch='70' \
            --with-cuda-home=$CUDA_HOME \
            --enable-cusparse \
            --enable-curand
make -j8 install
popd

cmake

You will need cmake to install some of the dependencies below and the Bridges2 native version is unsupported at the time of this writing. We’ll install a local one. You will need to set export CMAKE_INSTALL_DIR=$HOME/<path/to/my/cmake-install-dir> to a path that you have read/write access to.
```
wget https://cmake.org/files/v3.23/cmake-3.23.2.tar.gz
tar -zxvf cmake-3.23.2.tar.gz
pushd cmake-3.23.2
./bootstrap --prefix=$CMAKE_INSTALL_DIR
make -j8 install
popd
```
After installing cmake, you will need to prepend it to your path
```
export PATH=$CMAKE_INSTALL_DIR/bin:$PATH
```

Catch2

git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2
pushd Catch2/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

GMP

wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz
tar -xf gmp-6.2.1.tar.xz
pushd gmp-6.2.1
./configure --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

MPFR

wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz
tar -xf mpfr-4.1.0.tar.xz
pushd mpfr-4.1.0/
./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

CGAL

git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal
pushd cgal/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

PEGTL

git clone --branch 3.2.2 https://github.com/taocpp/PEGTL
pushd PEGTL/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

CSG EB library

Make sure you have boost loaded before building. (This is intended to be executed from inside the mfix repo.)

make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \
     PEGTL_HOME=$CSG_INSTALL_DIR \
     CGAL_HOME=$CSG_INSTALL_DIR \
     CATCH2_HOME=$CSG_INSTALL_DIR \
     ENABLE_CGAL=TRUE

Conduit

git clone --recursive https://github.com/LLNL/conduit.git
pushd conduit/
git checkout v0.8.6
mkdir build && cd build
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DENABLE_OPENMP=OFF \
      -DENABLE_MPI=ON \
      -DENABLE_CUDA=OFF \
      -DCMAKE_BUILD_TYPE=Release
make -j8 install
popd

Vtk-m

git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git
pushd vtk-m/
git checkout v1.9.0
mkdir build && cd build/
cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DVTKm_ENABLE_OPENMP=OFF \
      -DVTKm_ENABLE_MPI=ON \
      -DVTKm_ENABLE_CUDA=OFF \
      -DVTKm_USE_64BIT_IDS=OFF \
      -DVTKm_USE_DOUBLE_PRECISION=ON \
      -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \
      -DVTKm_NO_DEPRECATED_VIRTUAL=ON \
      -DCMAKE_BUILD_TYPE=Release
make -j8 install
popd

Ascent

git clone --recursive https://github.com/Alpine-DAV/ascent.git
pushd ascent
git checkout v0.9.0
mkdir build && cd build/
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
      -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \
      -DVTKM_DIR=$ASCENT_INSTALL_DIR \
      -DENABLE_VTKH=ON \
      -DENABLE_FORTRAN=OFF \
      -DENABLE_PYTHON=OFF \
      -DENABLE_DOCS=OFF \
      -DBUILD_SHARED_LIBS=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DENABLE_GTEST=OFF \
      -DENABLE_TESTS=OFF
make -j8 install
popd

Running Jobs

Common Slurm commands:

sinfo see available/allocated resources
sbatch runit_cpu.sh submit a cpu job to the queue
squeue -u USER check job status of user USER
squeue -p PARTITION check job status of partition PARTITION
scancel JOBID kill a job with id JOBID
salloc -N 1 -p GPU -A peb230001p -t 00:02:00 --exclusive --gpus-per-node=8 grab an entire GPU node for five minutes
salloc -N 1 -p GPU-shared -A peb230001p -t 00:05:00 --gpus-per-node=2 grab two GPUs interactively on a shared node for five minutes, you can grab up to four --gpus-per-node

Example run script for GPU is below. You can mirror the interactive commands above to convert this into a shared job script. CPU-only runs have not been tested on this machine.

#!/bin/bash
#!/bin/bash
#SBATCH -A peb230001p
#SBATCH -N 1
#SBATCH -p GPU-shared
#SBATCH -t 00:05:00
#SBATCH --gpus=v100-32:4

# load modules
module load gcc/10.2.0                 # PrgEnv
module load openmpi/4.0.5-gcc10.2.0    # MPI
module load cuda/11.7.1                # for GPU
module load boost/1.75-gcc10.2.0       # for CSG

#echo commands to stdout
set -x

mpirun -np 4 ./mfix inputs.rt > screen.txt