Installation instructions on Joule 3

jeff.dietiker · April 7, 2025, 2:08pm

Do you have a set of instructions to install Exa on Joule 3? The documentation has instructions for Joule 2 but Joule 3 doesn’t have the same modules (gnu/9.3.0, boost are not supported on Joule 3). Looks like Hypre doesn’t support gnu 14.2 (unsupported GNU version! gcc versions later than 13 are not supported!). I am not sure if I actually need to install Hypre. Do you have recommendation on when it becomes beneficial to use Hypre?

wfullmer · April 7, 2025, 3:13pm

Jeff, here are my personal notes for building on joule. They are pretty similar to my ubuntu install instructions. So someone pointed out that a dependency (I think boost) didn’t work anymore; you just need to bump that up to the latest version on their site. Also, as mentioned in the last user meeting, there is a problem w/ the latest hypre. You might be able to get a newer version than whatever I point to here (back when Joule3 came online), but don’t use the latest release. (Unless that’s been fixed that I’m unaware of.)

wfullmer · April 7, 2025, 3:14pm

Build all Dependencies on Joule3

comments

N/A

setup

Initialize the directory structure. I’m going to dump all the
MFIX-Exa dependencies in ~/packages/mfix-exa_deps-DATE/

export MY_INSTALL_DIR=$HOME/packages/mfix-exa-deps-24.06new
cd /nfs/scr/2/wfullmer/
rm -rf tmp.bld-deps/ 
mkdir tmp.bld-deps && cd tmp.bld-deps/

load modules

For CPU we’ll just build on the head node, go ahead and load modules and set your helpers in the terminal.

module purge
module load cmake/3.29.3
module load gnu/13.2.0
module load openmpi/4.1.5/gnu/13.2.0
export MPI_DIR=/nfs/apps/MPI/OpenMPI/4.1.5/gnu/13.2.0
#rm? export CC=$(which mpicc)
#rm? export CXX=$(which mpic++)
#rm? export F77=$(which mpif77)
#rm? export FC=$(which mpifort)
#rm? export F90=$(which mpif90)

complex geometry

export CSG_INSTALL_DIR=$MY_INSTALL_DIR/csg-deps
mkdir -p $CSG_INSTALL_DIR

Catch2

git clone --depth 1 --branch v3.6.0 https://github.com/catchorg/Catch2
pushd Catch2/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

GMP

wget --no-check-certificate https://gmplib.org/download/gmp/gmp-6.3.0.tar.xz
tar -xf gmp-6.3.0.tar.xz
pushd gmp-6.3.0
./configure --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

MPFR

wget --no-check-certificate https://www.mpfr.org/mpfr-current/mpfr-4.2.1.tar.gz
tar -xf mpfr-4.2.1.tar.xz
pushd mpfr-4.2.1/
./configure --with-gmp=$CSG_INSTALL_DIR --prefix=$CSG_INSTALL_DIR
make -j8 install
popd

CGAL

git clone --depth 1 --branch v5.6.1 https://github.com/CGAL/cgal
pushd cgal/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

PEGTL

git clone --branch 3.2.7 https://github.com/taocpp/PEGTL
pushd PEGTL/
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR
cd build/
make -j8 install
popd

boost

wget --no-check-certificate https://archives.boost.io/release/1.85.0/source/boost_1_85_0.tar.gz
tar -zxvf boost_1_85_0.tar.gz
pushd boost_1_85_0/
./bootstrap.sh
./b2 install --prefix=$CSG_INSTALL_DIR
popd

csg-eb

export CSG_LIB_DIR=$MY_INSTALL_DIR/csg-lib
mkdir $CSG_LIB_DIR

git clone https://mfix.netl.doe.gov/gitlab/exa/csg-eb.git
pushd csg-eb
make install DESTDIR=$CSG_LIB_DIR PEGTL_HOME=$CSG_INSTALL_DIR CGAL_HOME=$CSG_INSTALL_DIR CATCH2_HOME=$CSG_INSTALL_DIR ENABLE_CGAL=TRUE
popd

in situ vis

export ASCENT_INSTALL_DIR=$MY_INSTALL_DIR/ascent
mkdir $ASCENT_INSTALL_DIR

conduit

git clone --depth 1 --branch v0.8.8 --recursive https://github.com/LLNL/conduit.git
#git clone --depth 1 --branch v0.9.1 --recursive https://github.com/LLNL/conduit.git
pushd conduit/
mkdir build && cd build
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DENABLE_OPENMP=OFF -DENABLE_MPI=ON -DENABLE_CUDA=OFF 
make -j8 install
popd

vtk-m

git clone --depth 1 --branch v2.0.0 https://gitlab.kitware.com/vtk/vtk-m.git
#git clone --depth 1 --branch v2.1.0 https://gitlab.kitware.com/vtk/vtk-m.git
pushd vtk-m/
git checkout v2.0.0
#git checkout v2.1.0
mkdir build && cd build/
cmake -S ../ -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DVTKm_ENABLE_OPENMP=OFF -DVTKm_ENABLE_MPI=ON -DVTKm_ENABLE_CUDA=OFF -DVTKm_USE_64BIT_IDS=OFF -DVTKm_USE_DOUBLE_PRECISION=ON -DVTKm_USE_DEFAULT_TYPES_FOR_ASCENT=ON -DVTKm_NO_DEPRECATED_VIRTUAL=ON 
make -j8 install
popd

ascent

git clone --depth 1 --branch v0.9.2 --recursive https://github.com/Alpine-DAV/ascent.git
#git clone --depth 1 --branch v0.9.3 --recursive https://github.com/Alpine-DAV/ascent.git
pushd ascent
mkdir build && cd build/
cmake -S ../src -DCMAKE_INSTALL_PREFIX=$ASCENT_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release -DCONDUIT_DIR=$ASCENT_INSTALL_DIR -DVTKM_DIR=$ASCENT_INSTALL_DIR -DENABLE_VTKH=ON -DENABLE_FORTRAN=OFF -DENABLE_PYTHON=OFF -DENABLE_DOCS=OFF -DBUILD_SHARED_LIBS=ON
make -j8 install
popd

linear solvers

hypre cpu only

export HYPRE_INSTALL_DIR=$MY_INSTALL_DIR/hypre
mkdir $HYPRE_INSTALL_DIR 
#
git clone https://github.com/hypre-space/hypre.git
pushd hypre/src/
git checkout v2.31.0
./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI --with-MPI-lib-dirs="$MPI_DIR/lib" --with-MPI-include="$MPI_DIR/include"
make -j8 install 
popd

hypre gpu

module load cuda/12.4
unset HYPRE_INSTALL_DIR
export HYPRE_INSTALL_DIR=$MY_INSTALL_DIR/hypre-gpu
mkdir $HYPRE_INSTALL_DIR 
#
git clone https://github.com/hypre-space/hypre.git hypre2
pushd hypre2/src/
git checkout v2.31.0
./configure --prefix=$HYPRE_INSTALL_DIR --with-MPI --with-MPI-lib-dirs="$MPI_DIR/lib" --with-MPI-include="$MPI_DIR/include" --with-cuda --enable-shared --with-gpu-arch="90" --with-cuda-home=$CUDA_HOME --enable-cusparse --enable-curand
make -j8 install
popd

jeff.dietiker · April 7, 2025, 4:07pm

Thanks Will, I will give it a try!

wfullmer · April 8, 2025, 1:48pm

@jeff.dietiker I just realized these notes are just for the deps and stops short at building the code. Not that you couldn’t figure it out from the docs, but here are my joule3 build scripts which point to the dependencies above.

ok. well I can’t upload *.sh’s so here’s the cpu build script:

#!/bin/bash -l

## load the necessary modules
module purge
module load cmake/3.29.3
module load gnu/13.2.0
module load openmpi/4.1.5/gnu/13.2.0

## set env
export MY_INSTALL_DIR=$HOME/packages/mfix-exa-deps-24.06new
export HYPRE_DIR=$MY_INSTALL_DIR/hypre
export HYPRE_ROOT=$HYPRE_DIR
export HYPRE_LIBRARIES=$HYPRE_DIR/lib
export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include

export ASCENT_DIR=$MY_INSTALL_DIR/ascent
export CONDUIT_DIR=$ASCENT_DIR
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit

export CSG_DIR=$MY_INSTALL_DIR/csg-deps
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR
export Boost_INCLUDE_DIR="-I$CSG_INSTALL_DIR/include/boost"

## build mfix
cmake -DCMAKE_C_COMPILER=gcc \
      -DCMAKE_CXX_COMPILER=g++ \
      -DCMAKE_Fortran_COMPILER=gfortran \
      -DMFIX_MPI=yes \
      -DMFIX_OMP=no \
      -DMFIX_GPU_BACKEND=NONE \
      -DMFIX_CSG=yes \
      -DMFIX_GRANULAR=no \
      -DMFIX_HYPRE=yes \
      -DAMReX_ASCENT=yes \
      -DAMReX_CONDUIT=yes \
      -DAMReX_TINY_PROFILE=no \
      -DCMAKE_BUILD_TYPE=Release \
      ../mfix
make -j 8

## uncomment to build the pic2dem restarter app
#cmake -j8 --build . --target pic2dem

wait
exit
#eof

and gpu:

#!/bin/bash -l

##Load Modules
module purge
module load cmake/3.29.3
module load cuda/12.4
module load gnu/13.2.0
module load openmpi/4.1.5/gnu/13.2.0

## set env
export MY_INSTALL_DIR=$HOME/packages/mfix-exa-deps-24.06new
export HYPRE_DIR=$MY_INSTALL_DIR/hypre-gpu
export HYPRE_ROOT=$HYPRE_DIR
export HYPRE_LIBRARIES=$HYPRE_DIR/lib
export HYPRE_INCLUDE_DIRS=$HYPRE_DIR/include

export ASCENT_DIR=$MY_INSTALL_DIR/ascent
export CONDUIT_DIR=$ASCENT_DIR
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/ascent
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$ASCENT_DIR/lib/cmake/conduit

export CSG_DIR=$MY_INSTALL_DIR/csg-deps
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$CSG_DIR
export Boost_INCLUDE_DIR="-I$CSG_INSTALL_DIR/include/boost"

## build mfix
cmake -DCMAKE_C_COMPILER=gcc \
      -DCMAKE_CXX_COMPILER=g++ \
      -DCMAKE_Fortran_COMPILER=gfortran \
      -DBoost_INCLUDE_DIR="$Boost_INCLUDE_DIR" \
      -DMFIX_MPI=yes \
      -DMFIX_OMP=no \
      -DMFIX_GPU_BACKEND=CUDA \
      -DAMReX_CUDA_ARCH=90 \
      -DCMAKE_CUDA_ARCHITECTURES="90" \
      -DGPUS_PER_SOCKET=4 \
      -DGPUS_PER_NODE=8 \
      -DMFIX_CSG=yes \
      -DMFIX_GRANULAR=no \
      -DMFIX_HYPRE=yes \
      -DAMReX_ASCENT=yes \
      -DAMReX_CONDUIT=yes \
      -DAMReX_TINY_PROFILE=no \
      -DCMAKE_BUILD_TYPE=Release \
      ../mfix
make -j8

## uncomment to build the pic2dem restarter app
#cmake --build . --target pic2dem

wait
exit
#eof

jeff.dietiker · April 8, 2025, 8:18pm

Thanks Will, this is very useful! I was able to build and run the simple cylinder benchmark.

wakefiej · May 15, 2025, 6:10pm

@wfullmer This was very useful, thank you.

I don’t know how much interest you have in keeping this updated, but…

mpfr v4.2.1 is no longer the most recent version. The url for that version is now https://mpfr.org/mpfr-4.2.1/mpfr-4.2.1.tar.gz.
CGAL throws a warning: For performance reasons, you should set CMAKE_BUILD_TYPE to "Release".; this command should probably be $ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=$CSG_INSTALL_DIR -DCMAKE_BUILD_TYPE=Release.
I believe Boost_INCLUDE_DIR is supposed to be set by FindBoost.cmake—/.nfs/misc/apps/CMake/3.29.3/gnu/system/share/cmake-3.29/Modules/FindBoost.cmake on Joule—based on one of a few variables. I am not sure what the ideal way for this to work is, but I set export BOOST_ROOT="-I$CSG_DIR" instead and that seems to have worked.
It looks like GPUS_PER_NODE, GPUS_PER_SOCKET, and MFIX_GRANULAR are no longer used.

wfullmer · May 15, 2025, 6:32pm

Be careful updating versions