Commit 28c32549 authored by Ann Almgren's avatar Ann Almgren
Browse files

Still working on this...

parent 7773c437
Loading
Loading
Loading
Loading

docs/source/BuildingMFiX.rst

deleted100644 → 0
+0 −175
Original line number Diff line number Diff line
Building MFiX-Exa
=================

There are two options to building MFIX-Exa:

* SUPERBUILD (default):
    Utilities download and build AMReX as part of the MFIX-Exa build process.
    This method is strongly encouraged as it ensures configure options for
    MFIX-Exa and AMReX are consistent.

* Using an existing AMReX Library:
    MFIX-Exa is linked to an
    existing AMReX installation. This is ideal for continuous integration
    severs (CI) and regression testing applications.  AMReX library version
    must meet MFIX-Exa requirements.


Superbuild Instructions
-----------------------

When building in *SUPERBUILD* mode, MFiX-Exa build system will take care of
downloading, configuring and installing AMReX as part of the MFiX-Exa build.
The following commands build MFiX-Exa and AMReX in a single step.

.. highlight:: bash

::

    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    cd mfix
    mkdir build
    cd build
    cmake CONFIG_OPTIONS ..
    make -j

AMReX is built the first time the ``make`` command is issued.  No external
installation of AMReX is needed. However, internet access to the AMReX github
repository is required.  The optional string ``CONFIG_OPTIONS`` allows to
customize the build of both AMReX and MFIX-Exa.  ``CONFIG_OPTIONS`` is a list
of one or more configuration options given in the form ``-DOPTION=VALUE``.

The table below lists configuration options, possible values, and their effect
on the build.  Options prefixed by ``AMREX_`` are specific to the build of
AMReX.


.. table:: MFiX-Exa compiler options

    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | Option name                  |  Description                                 | Possible values              | Default value       |
    +==============================+==============================================+==============================+=====================+
    | DEBUG                        | Build in debug mode                          |   ON/OFF                     |   OFF               |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | MFIX_FFLAGS_OVERRIDES        | User-defined Fortran flags                   | valid F90 compiler flags     |   None              |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | MFIX_CXXFLAGS_OVERRIDES      | User-defined C++ flags                       | valid C++ compiler flags     |   None              |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | ENABLE_FPE                   | Build with Floating-Point Exceptions checks  |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_MPI             | Enable build with MPI                        |   0/1                        |   1                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_OMP             | Enable build with OpenMP                     |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_DP              | Enable double precision                      |   0/1                        |   1                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_DP_PARTICLES    | Enable double precision in particles classes |   0/1                        |   1                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_BASE_PROFILE    | Include profiling info                       |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_TINY_PROFILE    | Include tiny profiling info                  |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_COMM_PROFILE    | Include communicators profiling info         |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_TRACE_PROFILE   | Include trace profiling info                 |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_MEM_PROFILE     | Include memory profiling info                |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_BACKTRACE       | Include backtrace info                       |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_PROFPARSER      | Include profile parser                       |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_PIC             | Build position-independent code              |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_ENABLE_ASSERTION       | Build position-independent code              |   0/1                        |   0                 |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_GIT_COMMIT             | AMReX commit to be used in the build         | valid git commit id/branch   |   None              |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+
    | AMREX_INSTALL_DIR            | Global path to AMReX install directory       | valid global path            |   None (superbuild) |
    +------------------------------+----------------------------------------------+------------------------------+---------------------+

Note that SUPERBUILD mode is enabled automatically when ``AMREX_INSTALL_DIR``
is not given.

Example: build mfix with custom fortran flags, AMReX profiling enabled and
single precision particles:

.. highlight:: bash

::

    cmake -DMFIX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 ..


Building MFIX-Exa using a separate AMReX installation (no superbuild)
---------------------------------------------------------------------

Build AMReX Library
^^^^^^^^^^^^^^^^^^^

Clone AMReX from the official Git repository and checkout the _development_
branch.

.. highlight:: bash

::

    git clone https://github.com/AMReX-Codes/amrex.git
    cd amrex
    git checkout development

Next, configure, build and install AMReX as follows:

.. highlight:: bash

::

    cmake AMREX_CONFIG_OPTIONS -DENABLE_PARTICLES=1 -DENABLE_FBASELIB=1 -DENABLE_EB=1 -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir .
    make install

Here, ``AMREX_CONFIG_OPTIONS`` are optional configure options for AMReX. Please
refer to the AMReX user guide for a list of all the possible configuration
options. The only options required are ``-DENABLE_PARTICLES=1``,
``-DENABLE_FBASELIB=1``, and ``-DENABLE_EB=1``.


Building MFIX-Exa
^^^^^^^^^^^^^^^^^^

Clone and build MFIX-Exa.

.. highlight:: bash

::

    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    mkdir build
    cd build
    cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir ..
    make -j

Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options, that is,
any option NOT prefixed by ``AMREX_`` in the table above. Options prefixed by
``AMREX_`` are always ignored when using an external AMReX installation.


A Few more notes on building MFIX-Exa
-------------------------------------

The system defaults compilers can be overwritten as follows:

.. highlight:: bash

::

    cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> CONFIG_OPTIONS  ..

When building on a platform that uses the ``module`` utility, use either the above command (with full
path to the compilers) or the following:

.. highlight:: bash

::

    cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS  ..
+7 −54
Original line number Diff line number Diff line
MFiX-Exa Introduction
=====================

MFiX-Exa comes with its own Projection-Method based solver for the
Navier-Stokes equations, as well as the SIMPLE solver used by MFiX-Classic.
AMReX is used to manage the simulation's data. Hence, AMReX is also responsible
for MPI and openMP parallelization. In the future, AMReX's adaptive mesh
refinement (AMR) will also be supported by MFiX-Exa.

The code for now lives at NETL; instructions for how to get access to the code
will be given below.

Key features of MFiX-Exa include:

-  C++ and Fortran interfaces

-  Projection-Method based solver of the Nativer-Stokes Equations

-  Support for cell-centered, face-centered, edge-centered, and nodal data

-  Support for solid particles
.. _Chap:Introduction

-  Parallelization via flat MPI, OpenMP, hybrid MPI/OpenMP, or MPI/MPI

-  Parallel I/O

-  Plotfile format supported by AmrVis, VisIt, ParaView, and yt.

-  Complex container walls implemented as embedded boundaries, with efficient
   particle/wall interactions.

Besides this documentation, there is documentation generated by Doxygen at
https://amrex-codes.github.io/MFIX-Exa/doxygen.
MFiX-Exa Introduction
==================

.. toctree::
   :maxdepth: 1
   :caption: Contents:

   BuildingMFiX
   TestingMFiX


Directory overview
------------------

.. table:: Overview of the MFiX-Exa directory structure
   Structure
   Building
   RunningTestSuite
   RegressionTesting
    +------------+-----------------------------------------------------+
    | File       | Description                                         |
    +============+=====================================================+
    | benchmarks | UC Benchmark cases (see benchmark/README.md)        |
    +------------+-----------------------------------------------------+
    | src        | Fortran source files                                |
    +------------+-----------------------------------------------------+
    | tests      | Regression tests (see tests/README.md)              |
    +------------+-----------------------------------------------------+
    | ThirdParty | External libraries sources                          |
    +------------+-----------------------------------------------------+
    | Tools      | CMake configuration files                           |
    +------------+-----------------------------------------------------+
+0 −14
Original line number Diff line number Diff line
@@ -86,17 +86,3 @@ Run a user-defined case
this can be named anything as long as it is the **first** argument
following the executable. Note that many of the problem parameters are
still defined in *mfix.dat*.

See the User's Guide for more about MFiX-Exa
============================================

To build the User's Guide,

.. code:: shell

    > cd doc/UsersGuide
    > make

This will build a pdf of the MFiX-Exa User's Guide, which contains
information about the equations being solved, run-time parameters,
checkpoint/restart capability, options for visualization and more.

docs/source/TestingMFiX.rst

deleted100644 → 0
+0 −206
Original line number Diff line number Diff line
Running the MFIX Test Suite
===========================

MFiX-Exa comes with several tests aimed at evaluating software functionalities.
The source files as well as the required input files for each test are located
in the ``tests`` directory. The ``tests`` directory is copied to the build
directory during MFIX-Exa configuration process. When a test is run (see
below), output files are stored in ``build_dir/tests/test-name``.

There are various dependencies for comparing test results.

* To compare results to archived flow slices stored in ``AUTOTEST``
  directories with the case files, the environment variable ``FEXTRACT``
  must point to the location of the AMReX `fextract` utility located
  in the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally,
  ``numdiff`` must be installed for comparing results.

* To compare point-by-point field data, the environment variable
  ``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids``
  found in the directory, ``amrex/Tools/PostProcessing/F_Src``.
  Additionally, the environment variable ``MFIX_BENCHMARKS_HOME``
  must point to the location of a local regression test data set.
  See :ref:`sec:test:generate_data` for instructions on
  creating a local regression test data set.


Run all tests
^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir
    ctest


List all tests (without running them)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir
    ctest -N


Run a particular test by the index listed in ``ctest -N``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir
    ctest -I 3,3             # run the third test


Run a particular test by name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir
    ctest -R DEM01  # running all tests with "DEM01" in the test name


Run a particular test via make
------------------------------

.. highlight:: bash

::

    cd to mfix-build-dir
    make run_DEM01-x  # running "DEM01-x" and output to the screen


Run specific
^^^^^^^^^^^^

If the environment variable GRID is defined, it specifies which grid types to
run for the test(s).  If GRID variable is not defined, the default is to run
the tests for all grid types.

.. highlight:: bash

::

    env GRID="tiled" ctest -R DEM01  # running all tests with "DEM01" for tiled grid
    env GRID="single multiple" ctest -R DEM01  # running all tests with "DEM01" for single grid and multiple grid
    ctest -R DEM01  # running all tests with "DEM01" for all grid types (single, multiple, tiled)


Run a user-defined case
^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    ./mfix inputs-myrun

``inputs-myrun`` is a text file containing the AMReX input parameters; this can
be named anything as long as it is the **first** argument following the
executable. Note that many of the problem parameters are still defined in
``mfix.dat``.


Regression testing
------------------

.. _sec:test:generate_data:

Generating local regression test data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Developers are encouraged to create local benchmark data for regression testing
prior to committing code the GitLab repository.

1. Create a location to store benchmark data and clone the MFiX and AMReX
   repositories.

.. highlight:: bash

::

    mkdir /home/user/exa-rt
    mkdir /home/user/exa-rt/benchmark
    cd /home/user/exa-rt
    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
    git clone https://github.com/AMReX-Codes/amrex.git


2. Create a local copy the regression test setup file from the MFiX repository.

.. highlight:: bash

::

    cp mfix/RegressionTesting/MFIX-tests.ini MFIX-local.ini


3. Edit the local setup file. Specify the top level directories for test and
   web output under the ``[main]`` heading.

.. highlight:: ini

::

    [main]
    testTopDir = /home/user/exa-rt/benchmark
    webTopDir  = /home/user/exa-rt/web


Specify the AMReX source directory location under the ``[AMReX]`` heading.

.. highlight:: ini

::

    [AMReX]
    dir    = /home/user/exa-rt/amrex
    branch = development


Specify the MFIX-Exa source directory location under the ``[source]`` heading.

.. highlight:: ini

::

    [source]
    dir    = /home/user/exa-rt/mfix
    branch = develop


4. Run the AMReX regression test tool. The second argument is a user supplied
   comment.

.. highlight:: bash

::

    cd /home/user/exa-rt
    amrex/Tools/RegressionTesting/regtest.py --make_benchmarks "MFIX" MFIX-local.ini


Prerequisite: Environment Dependencies on Joule (Joule specific)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For the Joule environment, load the gnu module and set environment variables
first. If not on Joule, skip this step.

.. highlight:: bash

::

    module load gnu/6.1.0
    export CC=/nfs/apps/Compilers/GNU/6.1.0/bin/gcc
    export CXX=/nfs/apps/Compilers/GNU/6.1.0/bin/g++
    export F77=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran
    export FC=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran
+1 −4
Original line number Diff line number Diff line
@@ -20,10 +20,7 @@ the master branch at the beginning of each month.
   :maxdepth: 2
   :caption: Contents:

   Structure
   Building
   RunningTestSuite
   RegressionTesting
   Introduction

Notice
------