initialize skeleton documentation based on README

docs/source/.mypy_cache

docs/build

# Minimal makefile for Sphinx documentation

#

# You can set these variables from the command line.

SPHINXOPTS    =

SPHINXBUILD   = python -msphinx

SPHINXPROJ    = amrex

SOURCEDIR     = source

BUILDDIR      = build

# Put it first so that "make" without argument is like "make help".

help:

	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new

# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

%: Makefile

	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

 No newline at end of file

Building MFiX-Exa

=================

There are two options to building MFIX-Exa:

* SUPERBUILD (default):

    Utilities download and build AMReX as part of the MFIX-Exa build process.

    This method is strongly encouraged as it ensures configure options for

    MFIX-Exa and AMReX are consistent.

* Using an existing AMReX Library:

    MFIX-Exa is linked to an

    existing AMReX installation. This is ideal for continuous integration

    severs (CI) and regression testing applications.  AMReX library version

    must meet MFIX-Exa requirements.

Superbuild Instructions

-----------------------

When building in *SUPERBUILD* mode, MFiX-Exa build system will take care of

downloading, configuring and installing AMReX as part of the MFiX-Exa build.

The following commands build MFiX-Exa and AMReX in a single step.

.. highlight:: bash

::

    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

    cd mfix

    mkdir build

    cd build

    cmake CONFIG_OPTIONS ..

    make -j

AMReX is built the first time the ``make`` command is issued.  No external

installation of AMReX is needed. However, internet access to the AMReX github

repository is required.  The optional string ``CONFIG_OPTIONS`` allows to

customize the build of both AMReX and MFIX-Exa.  ``CONFIG_OPTIONS`` is a list

of one or more configuration options given in the form ``-DOPTION=VALUE``.

The table below lists configuration options, possible values, and their effect

on the build.  Options prefixed by ``AMREX_`` are specific to the build of

AMReX.

.. table:: MFiX-Exa compiler options

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | Option name                  |  Description                                 | Possible values              | Default value       |

    +==============================+==============================================+==============================+=====================+

    | DEBUG                        | Build in debug mode                          |   ON/OFF                     |   OFF               |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | MFIX_FFLAGS_OVERRIDES        | User-defined Fortran flags                   | valid F90 compiler flags     |   None              |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | MFIX_CXXFLAGS_OVERRIDES      | User-defined C++ flags                       | valid C++ compiler flags     |   None              |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | ENABLE_FPE                   | Build with Floating-Point Exceptions checks  |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_MPI             | Enable build with MPI                        |   0/1                        |   1                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_OMP             | Enable build with OpenMP                     |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_DP              | Enable double precision                      |   0/1                        |   1                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_DP_PARTICLES    | Enable double precision in particles classes |   0/1                        |   1                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_BASE_PROFILE    | Include profiling info                       |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_TINY_PROFILE    | Include tiny profiling info                  |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_COMM_PROFILE    | Include communicators profiling info         |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_TRACE_PROFILE   | Include trace profiling info                 |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_MEM_PROFILE     | Include memory profiling info                |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_BACKTRACE       | Include backtrace info                       |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_PROFPARSER      | Include profile parser                       |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_PIC             | Build position-independent code              |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_ENABLE_ASSERTION       | Build position-independent code              |   0/1                        |   0                 |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_GIT_COMMIT             | AMReX commit to be used in the build         | valid git commit id/branch   |   None              |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

    | AMREX_INSTALL_DIR            | Global path to AMReX install directory       | valid global path            |   None (superbuild) |

    +------------------------------+----------------------------------------------+------------------------------+---------------------+

Note that SUPERBUILD mode is enabled automatically when ``AMREX_INSTALL_DIR``

is not given.

Example: build mfix with custom fortran flags, AMReX profiling enabled and

single precision particles:

.. highlight:: bash

::

    cmake -DMFIX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 ..

Building MFIX-Exa using a separate AMReX installation (no superbuild)

---------------------------------------------------------------------

Build AMReX Library

^^^^^^^^^^^^^^^^^^^

Clone AMReX from the official Git repository and checkout the _development_

branch.

.. highlight:: bash

::

    git clone https://github.com/AMReX-Codes/amrex.git

    cd amrex

    git checkout development

Next, configure, build and install AMReX as follows:

.. highlight:: bash

::

    cmake AMREX_CONFIG_OPTIONS -DENABLE_PARTICLES=1 -DENABLE_FBASELIB=1 -DENABLE_EB=1 -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir .

    make install

Here, ``AMREX_CONFIG_OPTIONS`` are optional configure options for AMReX. Please

refer to the AMReX user guide for a list of all the possible configuration

options. The only options required are ``-DENABLE_PARTICLES=1``,

``-DENABLE_FBASELIB=1``, and ``-DENABLE_EB=1``.

Building MFIX-Exa

^^^^^^^^^^^^^^^^^^

Clone and build MFIX-Exa.

.. highlight:: bash

::

    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

    mkdir build

    cd build

    cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir ..

    make -j

Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options, that is,

any option NOT prefixed by ``AMREX_`` in the table above. Options prefixed by

``AMREX_`` are always ignored when using an external AMReX installation.

A Few more notes on building MFIX-Exa

-------------------------------------

The system defaults compilers can be overwritten as follows:

.. highlight:: bash

::

    cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> CONFIG_OPTIONS  ..

When building on a platform that uses the ``module`` utility, use either the above command (with full

path to the compilers) or the following:

.. highlight:: bash

::

    cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS  ..

MFiX-Exa Introduction

=====================

MFiX-Exa comes with its own Projection-Method based solver for the

Navier-Stokes equations, as well as the SIMPLE solver used by MFiX-Classic.

AMReX is used to manage the simulation's data. Hence, AMReX is also responsible

for MPI and openMP parallelization. In the future, AMReX's adaptive mesh

refinement (AMR) will also be supported by MFiX-Exa.

The code for now lives at NETL; instructions for how to get access to the code

will be given below.

Key features of MFiX-Exa include:

-  C++ and Fortran interfaces

-  Projection-Method based solver of the Nativer-Stokes Equations

-  Support for cell-centered, face-centered, edge-centered, and nodal data

-  Support for solid particles

-  Parallelization via flat MPI, OpenMP, hybrid MPI/OpenMP, or MPI/MPI

-  Parallel I/O

-  Plotfile format supported by AmrVis, VisIt, ParaView, and yt.

-  Complex container walls implemented as embedded boundaries, with efficient

   particle/wall interactions.

Besides this documentation, there is documentation generated by Doxygen at

https://amrex-codes.github.io/amrex/doxygen. Documentation on migration from

BoxLib is available at Docs/Migration.

.. toctree::

   :maxdepth: 1

   :caption: Contents:

   BuildingMFiX

   TestingMFiX

Directory overview

------------------

.. table:: Overview of the MFiX-Exa directory structure

    +------------+-----------------------------------------------------+

    | File       | Description                                         |

    +============+=====================================================+

    | benchmarks | UC Benchmark cases (see benchmark/README.md)        |

    +------------+-----------------------------------------------------+

    | src        | Fortran source files                                |

    +------------+-----------------------------------------------------+

    | tests      | Regression tests (see tests/README.md)              |

    +------------+-----------------------------------------------------+

    | ThirdParty | External libraries sources                          |

    +------------+-----------------------------------------------------+

    | Tools      | CMake configuration files                           |

    +------------+-----------------------------------------------------+

Running the MFIX Test Suite

===========================

MFiX-Exa comes with several tests aimed at evaluating software functionalities.

The source files as well as the required input files for each test are located

in the ``tests`` directory. The ``tests`` directory is copied to the build

directory during MFIX-Exa configuration process. When a test is run (see

below), output files are stored in ``build_dir/tests/test-name``.

There are various dependencies for comparing test results.

* To compare results to archived flow slices stored in ``AUTOTEST``

  directories with the case files, the environment variable ``FEXTRACT``

  must point to the location of the AMReX `fextract` utility located

  in the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally,

  ``numdiff`` must be installed for comparing results.

* To compare point-by-point field data, the environment variable

  ``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids``

  found in the directory, ``amrex/Tools/PostProcessing/F_Src``.

  Additionally, the environment variable ``MFIX_BENCHMARKS_HOME``

  must point to the location of a local regression test data set.

  See :ref:`sec:test:generate_data` for instructions on

  creating a local regression test data set.

Run all tests

^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir

    ctest

List all tests (without running them)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir

    ctest -N

Run a particular test by the index listed in ``ctest -N``

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir

    ctest -I 3,3             # run the third test

Run a particular test by name

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    cd to mfix-build-dir

    ctest -R DEM01  # running all tests with "DEM01" in the test name

Run a particular test via make

------------------------------

.. highlight:: bash

::

    cd to mfix-build-dir

    make run_DEM01-x  # running "DEM01-x" and output to the screen

Run specific

^^^^^^^^^^^^

If the environment variable GRID is defined, it specifies which grid types to

run for the test(s).  If GRID variable is not defined, the default is to run

the tests for all grid types.

.. highlight:: bash

::

    env GRID="tiled" ctest -R DEM01  # running all tests with "DEM01" for tiled grid

    env GRID="single multiple" ctest -R DEM01  # running all tests with "DEM01" for single grid and multiple grid

    ctest -R DEM01  # running all tests with "DEM01" for all grid types (single, multiple, tiled)

Run a user-defined case

^^^^^^^^^^^^^^^^^^^^^^^

.. highlight:: bash

::

    ./mfix inputs-myrun

``inputs-myrun`` is a text file containing the AMReX input parameters; this can

be named anything as long as it is the **first** argument following the

executable. Note that many of the problem parameters are still defined in

``mfix.dat``.

Regression testing

------------------

.. _sec:test:generate_data:

Generating local regression test data

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Developers are encouraged to create local benchmark data for regression testing

prior to committing code the GitLab repository.

1. Create a location to store benchmark data and clone the MFiX and AMReX

   repositories.

.. highlight:: bash

::

    mkdir /home/user/exa-rt

    mkdir /home/user/exa-rt/benchmark

    cd /home/user/exa-rt

    git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

    git clone https://github.com/AMReX-Codes/amrex.git

2. Create a local copy the regression test setup file from the MFiX repository.

.. highlight:: bash

::

    cp mfix/RegressionTesting/MFIX-tests.ini MFIX-local.ini

3. Edit the local setup file. Specify the top level directories for test and

   web output under the ``[main]`` heading.

.. highlight:: ini

::

    [main]

    testTopDir = /home/user/exa-rt/benchmark

    webTopDir  = /home/user/exa-rt/web

Specify the AMReX source directory location under the ``[AMReX]`` heading.

.. highlight:: ini

::

    [AMReX]

    dir    = /home/user/exa-rt/amrex

    branch = development

Specify the MFIX-Exa source directory location under the ``[source]`` heading.

.. highlight:: ini

::

    [source]

    dir    = /home/user/exa-rt/mfix

    branch = develop

4. Run the AMReX regression test tool. The second argument is a user supplied

   comment.

.. highlight:: bash

::

    cd /home/user/exa-rt

    amrex/Tools/RegressionTesting/regtest.py --make_benchmarks "MFIX" MFIX-local.ini

Prerequisite: Environment Dependencies on Joule (Joule specific)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For the Joule environment, load the gnu module and set environment variables

first. If not on Joule, skip this step.

.. highlight:: bash

::

    module load gnu/6.1.0

    export CC=/nfs/apps/Compilers/GNU/6.1.0/bin/gcc

    export CXX=/nfs/apps/Compilers/GNU/6.1.0/bin/g++

    export F77=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran

    export FC=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran