# PSC Bridges-2

If this is your first time building MFIX-Exa on Bridges-2, please review the general notes below and Basics section first.

- To access Delta-2, you will need PSC account which can be administered by the NSF ACCESS allocation management system.

- You can find the name of your account(s) and the current balance with the `accounts` command.

- These instructions build MFIX-Exa on the login nodes using -j8 CPUs. You may have to decrease this value if there is high traffic or you may want to increase this value if you are on a compute node interactively.

- For the dependencies, it is assumed that you have set the following environment variables:

      export HYPRE_INSTALL_DIR=$HOME/<path/to/my/hypre-install-dir>
      export CSG_INSTALL_DIR=$HOME/<path/to/my/csg-dep-install-dir>
      export CSG_LIB_DIR=$HOME/<path/to/my/csg-lib-install-dir>
      export ASCENT_INSTALL_DIR=$HOME/<path/to/my/ascent-install-dir>

  to a path that you have read/write access to. You will need to recall these paths later if you want to build MFIX-Exa with the optional dependencies.

## Basics

### Source code

Before building, first obtain a copy of the source code following the instructions on the MFIX-Exa website.

### Modules

There are (basically) no modules loaded by default on Bridges-2.

    module load gcc/10.2.0                 # PrgEnv
    module load openmpi/4.0.5-gcc10.2.0    # MPI
    module load cuda/11.7.1                # for GPU
    module load boost/1.75-gcc10.2.0       # for CSG

## Building MFIX-Exa

CMake 3.20 or higher is required. You are running version 3.11.4

The commands below are the superbuild instructions, i.e., AMReX is built as part of the MFIX-Exa build process. To build MFIX-Exa with hypre, csg and/or ascent dependencies, you first need to build and install these libraries and their dependencies. Instructions on building the necessary dependencies are below and should be successfully installed first. Only the `gmake` method of building MFIX-Exa is currently recommended.

    cp -r exec exec.cpu
    make -C exec.cpu -j8 \
         COMP=gnu \
         USE_MPI=TRUE \
         USE_OMP=FALSE \
         USE_CUDA=FALSE \
         USE_TINY_PROFILE=FALSE \
         USE_CSG=FALSE \
         USE_HYPRE=FALSE \
         DEBUG=FALSE

    cp -r exec exec.gpu
    make -C exec.gpu -j8 \
         COMP=gnu \
         USE_MPI=TRUE \
         USE_OMP=FALSE \
         USE_CUDA=TRUE \
         CUDA_ARCH=70 \
         USE_TINY_PROFILE=FALSE \
         USE_CSG=FALSE \
         USE_HYPRE=FALSE \
         DEBUG=FALSE

    export HYPRE_DIR=$HYPRE_INSTALL_DIR
    export HYPRE_HOME=$HYPRE_DIR

    export ASCENT_DIR=$ASCENT_INSTALL_DIR
    export CONDUIT_DIR=$ASCENT_DIR

    export CSGEB_HOME=$CSG_LIB_DIR
    export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib"
    cp -r exec exec.cpu.full
    make -C exec.cpu.full -j8 \
         COMP=gnu \
         USE_MPI=TRUE \
         USE_OMP=FALSE \
         USE_CUDA=FALSE \
         USE_TINY_PROFILE=FALSE \
         USE_CSG=TRUE \
         USE_HYPRE=TRUE \
         USE_ASCENT=TRUE \
         USE_CONDUIT=TRUE \
         DEBUG=FALSE

    export HYPRE_DIR=$HYPRE_INSTALL_DIR
    export HYPRE_HOME=$HYPRE_DIR

    export ASCENT_DIR=$ASCENT_INSTALL_DIR
    export CONDUIT_DIR=$ASCENT_DIR

    export CSGEB_HOME=$CSG_LIB_DIR
    export LDFLAGS="-lgmp -lmpfr -L$CSG_INSTALL_DIR/lib -Wl,-rpath=$CSG_INSTALL_DIR/lib"
    cp -r exec exec.gpu.full
    make -C exec.gpu.full -j8 COMP=gnu \
         USE_MPI=TRUE \
         USE_OMP=FALSE \
         USE_CUDA=TRUE \
         CUDA_ARCH=70 \
         USE_TINY_PROFILE=FALSE \
         USE_CSG=TRUE \
         USE_HYPRE=TRUE \
         USE_ASCENT=TRUE \
         USE_CONDUIT=TRUE \
         DEBUG=FALSE

## Optional build dependencies

The following dependencies need to be built and installed prior to following any of the full build instructions above.

1. HYPRE

       git clone https://github.com/hypre-space/hypre.git
       pushd hypre/src/
       git checkout v2.26.0
       ./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI
       make -j8 install
       popd

       git clone https://github.com/hypre-space/hypre.git
       pushd hypre/src/
       git checkout v2.26.0
       ./configure --prefix=$HYPRE_INSTALL_DIR \
                   --without-superlu \
                   --disable-bigint \
                   --without-openmp \
                   --enable-unified-memory \
                   --with-MPI \
                   --with-cuda \
                   --with-gpu-arch='70' \
                   --with-cuda-home=$CUDA_HOME \
                   --enable-cusparse \
                   --enable-curand
       make -j8 install
       popd

2. cmake

   You will need cmake to install some of the dependencies below and the Bridges2 native version is unsupported at the time of this writing. We’ll install a local one. You will need to set `export CMAKE_INSTALL_DIR=$HOME/<path/to/my/cmake-install-dir>` to a path that you have read/write access to.

       wget https://cmake.org/files/v3.23/cmake-3.23.2.tar.gz
       tar -zxvf cmake-3.23.2.tar.gz
       pushd cmake-3.23.2
       ./bootstrap --prefix=$CMAKE_INSTALL_DIR
       make -j8 install
       popd

   After installing cmake, you will need to prepend it to your path

       export PATH=$CMAKE_INSTALL_DIR/bin:$PATH

3. Catch2

       git clone --depth 1 --branch v2.13.7 https://github.com/catchorg/Catch2
       pushd Catch2/
       cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
       cd build/
       make -j8 install
       popd

4. GMP

       wget --no-check-certificate https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz
       tar -xf gmp-6.2.1.tar.xz
       pushd gmp-6.2.1
       ./configure --prefix=$CSG_INSTALL_DIR
       make -j8 install
       popd

5. MPFR

       wget --no-check-certificate https://ftp.gnu.org/gnu/mpfr/mpfr-4.1.0.tar.xz
       tar -xf mpfr-4.1.0.tar.xz
       pushd mpfr-4.1.0/
       ./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR
       make -j8 install
       popd

6. CGAL

       git clone --depth 1 --branch v5.3 https://github.com/CGAL/cgal
       pushd cgal/
       cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
       cd build/
       make -j8 install
       popd

7. PEGTL

       git clone --branch 3.2.2 https://github.com/taocpp/PEGTL
       pushd PEGTL/
       cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
       cd build/
       make -j8 install
       popd

8. CSG EB library

   Make sure you have boost loaded before building. (This is intended to be executed from inside the mfix repo.)

       make -C subprojects/csg-eb install DESTDIR=$CSG_LIB_DIR \
            PEGTL_HOME=$CSG_INSTALL_DIR \
            CGAL_HOME=$CSG_INSTALL_DIR \
            CATCH2_HOME=$CSG_INSTALL_DIR \
            ENABLE_CGAL=TRUE

9. Conduit

       git clone --recursive https://github.com/LLNL/conduit.git
       pushd conduit/
       git checkout v0.8.6
       mkdir build && cd build
       cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
             -DENABLE_OPENMP=OFF \
             -DENABLE_MPI=ON \
             -DENABLE_CUDA=OFF \
             -DCMAKE_BUILD_TYPE=Release
       make -j8 install
       popd

10. Vtk-m

        git clone --branch master https://gitlab.kitware.com/vtk/vtk-m.git
        pushd vtk-m/
        git checkout v1.9.0
        mkdir build && cd build/
        cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
              -DVTKm_ENABLE_OPENMP=OFF \
              -DVTKm_ENABLE_MPI=ON \
              -DVTKm_ENABLE_CUDA=OFF \
              -DVTKm_USE_64BIT_IDS=OFF \
              -DVTKm_USE_DOUBLE_PRECISION=ON \
              -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON \
              -DVTKm_NO_DEPRECATED_VIRTUAL=ON \
              -DCMAKE_BUILD_TYPE=Release
        make -j8 install
        popd

11. Ascent

        git clone --recursive https://github.com/Alpine-DAV/ascent.git
        pushd ascent
        git checkout v0.9.0
        mkdir build && cd build/
        cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR \
              -DCONDUIT_DIR=$ASCENT_INSTALL_DIR \
              -DVTKM_DIR=$ASCENT_INSTALL_DIR \
              -DENABLE_VTKH=ON \
              -DENABLE_FORTRAN=OFF \
              -DENABLE_PYTHON=OFF \
              -DENABLE_DOCS=OFF \
              -DBUILD_SHARED_LIBS=ON \
              -DCMAKE_BUILD_TYPE=Release \
              -DENABLE_GTEST=OFF \
              -DENABLE_TESTS=OFF
        make -j8 install
        popd

## Running Jobs

Common Slurm commands:

- `sinfo` see available/allocated resources

- `sbatch runit_cpu.sh` submit a cpu job to the queue

- `squeue -u USER` check job status of user USER

- `squeue -p PARTITION` check job status of partition PARTITION

- `scancel JOBID` kill a job with id JOBID

- `salloc -N 1 -p GPU -A peb230001p -t 00:02:00 --exclusive --gpus-per-node=8` grab an entire GPU node for five minutes

- `salloc -N 1 -p GPU-shared -A peb230001p -t 00:05:00 --gpus-per-node=2` grab two GPUs interactively on a shared node for five minutes, you can grab up to four `--gpus-per-node`

Example run script for GPU is below. You can mirror the interactive commands above to convert this into a shared job script. CPU-only runs have not been tested on this machine.

    #!/bin/bash
    #!/bin/bash
    #SBATCH -A peb230001p
    #SBATCH -N 1
    #SBATCH -p GPU-shared
    #SBATCH -t 00:05:00
    #SBATCH --gpus=v100-32:4

    # load modules
    module load gcc/10.2.0                 # PrgEnv
    module load openmpi/4.0.5-gcc10.2.0    # MPI
    module load cuda/11.7.1                # for GPU
    module load boost/1.75-gcc10.2.0       # for CSG

    #echo commands to stdout
    set -x

    mpirun -np 4 ./mfix inputs.rt > screen.txt