.gitignore 0 → 100644
docs/source/.mypy_cache
docs/build
docs/Makefile 0 → 100644
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = python -msphinx
SPHINXPROJ = amrex
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
Building MFiX-Exa
=================
There are two options to building MFIX-Exa:
* SUPERBUILD (default):
Utilities download and build AMReX as part of the MFIX-Exa build process.
This method is strongly encouraged as it ensures configure options for
MFIX-Exa and AMReX are consistent.
* Using an existing AMReX Library:
MFIX-Exa is linked to an
existing AMReX installation. This is ideal for continuous integration
severs (CI) and regression testing applications. AMReX library version
must meet MFIX-Exa requirements.
Superbuild Instructions
-----------------------
When building in *SUPERBUILD* mode, MFiX-Exa build system will take care of
downloading, configuring and installing AMReX as part of the MFiX-Exa build.
The following commands build MFiX-Exa and AMReX in a single step.
.. highlight:: bash
::
git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
cd mfix
mkdir build
cd build
cmake CONFIG_OPTIONS ..
make -j
AMReX is built the first time the ``make`` command is issued. No external
installation of AMReX is needed. However, internet access to the AMReX github
repository is required. The optional string ``CONFIG_OPTIONS`` allows to
customize the build of both AMReX and MFIX-Exa. ``CONFIG_OPTIONS`` is a list
of one or more configuration options given in the form ``-DOPTION=VALUE``.
The table below lists configuration options, possible values, and their effect
on the build. Options prefixed by ``AMREX_`` are specific to the build of
AMReX.
.. table:: MFiX-Exa compiler options
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| Option name | Description | Possible values | Default value |
+==============================+==============================================+==============================+=====================+
| DEBUG | Build in debug mode | ON/OFF | OFF |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| MFIX_FFLAGS_OVERRIDES | User-defined Fortran flags | valid F90 compiler flags | None |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| MFIX_CXXFLAGS_OVERRIDES | User-defined C++ flags | valid C++ compiler flags | None |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| ENABLE_FPE | Build with Floating-Point Exceptions checks | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_MPI | Enable build with MPI | 0/1 | 1 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_OMP | Enable build with OpenMP | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_DP | Enable double precision | 0/1 | 1 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_DP_PARTICLES | Enable double precision in particles classes | 0/1 | 1 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_BASE_PROFILE | Include profiling info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_TINY_PROFILE | Include tiny profiling info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_COMM_PROFILE | Include communicators profiling info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_TRACE_PROFILE | Include trace profiling info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_MEM_PROFILE | Include memory profiling info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_BACKTRACE | Include backtrace info | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_PROFPARSER | Include profile parser | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_PIC | Build position-independent code | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_ENABLE_ASSERTION | Build position-independent code | 0/1 | 0 |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_GIT_COMMIT | AMReX commit to be used in the build | valid git commit id/branch | None |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
| AMREX_INSTALL_DIR | Global path to AMReX install directory | valid global path | None (superbuild) |
+------------------------------+----------------------------------------------+------------------------------+---------------------+
Note that SUPERBUILD mode is enabled automatically when ``AMREX_INSTALL_DIR``
is not given.
Example: build mfix with custom fortran flags, AMReX profiling enabled and
single precision particles:
.. highlight:: bash
::
cmake -DMFIX_FFLAGS_OVERRIDES="custom flags" -DAMREX_ENABLE_BASE_PROFILE=1 -DAMREX_ENABLE_DP_PARTICLES=0 ..
Building MFIX-Exa using a separate AMReX installation (no superbuild)
---------------------------------------------------------------------
Build AMReX Library
^^^^^^^^^^^^^^^^^^^
Clone AMReX from the official Git repository and checkout the _development_
branch.
.. highlight:: bash
::
git clone https://github.com/AMReX-Codes/amrex.git
cd amrex
git checkout development
Next, configure, build and install AMReX as follows:
.. highlight:: bash
::
cmake AMREX_CONFIG_OPTIONS -DENABLE_PARTICLES=1 -DENABLE_FBASELIB=1 -DENABLE_EB=1 -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/installdir .
make install
Here, ``AMREX_CONFIG_OPTIONS`` are optional configure options for AMReX. Please
refer to the AMReX user guide for a list of all the possible configuration
options. The only options required are ``-DENABLE_PARTICLES=1``,
``-DENABLE_FBASELIB=1``, and ``-DENABLE_EB=1``.
Building MFIX-Exa
^^^^^^^^^^^^^^^^^^
Clone and build MFIX-Exa.
.. highlight:: bash
::
git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
mkdir build
cd build
cmake CONFIG_OPTIONS -DAMREX_INSTALL_DIR=/absolute/path/to/amrex/installdir ..
make -j
Here, ``CONFIG_OPTIONS`` are MFiX-Exa specific configuration options, that is,
any option NOT prefixed by ``AMREX_`` in the table above. Options prefixed by
``AMREX_`` are always ignored when using an external AMReX installation.
A Few more notes on building MFIX-Exa
-------------------------------------
The system defaults compilers can be overwritten as follows:
.. highlight:: bash
::
cmake -DCMAKE_CXX_COMPILER=<c++-compiler> -DCMAKE_Fortran_COMPILER=<f90-compiler> CONFIG_OPTIONS ..
When building on a platform that uses the ``module`` utility, use either the above command (with full
path to the compilers) or the following:
.. highlight:: bash
::
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn CONFIG_OPTIONS ..
MFiX-Exa Introduction
=====================
MFiX-Exa comes with its own Projection-Method based solver for the
Navier-Stokes equations, as well as the SIMPLE solver used by MFiX-Classic.
AMReX is used to manage the simulation's data. Hence, AMReX is also responsible
for MPI and openMP parallelization. In the future, AMReX's adaptive mesh
refinement (AMR) will also be supported by MFiX-Exa.
The code for now lives at NETL; instructions for how to get access to the code
will be given below.
Key features of MFiX-Exa include:
- C++ and Fortran interfaces
- Projection-Method based solver of the Nativer-Stokes Equations
- Support for cell-centered, face-centered, edge-centered, and nodal data
- Support for solid particles
- Parallelization via flat MPI, OpenMP, hybrid MPI/OpenMP, or MPI/MPI
- Parallel I/O
- Plotfile format supported by AmrVis, VisIt, ParaView, and yt.
- Complex container walls implemented as embedded boundaries, with efficient
particle/wall interactions.
Besides this documentation, there is documentation generated by Doxygen at
https://amrex-codes.github.io/amrex/doxygen. Documentation on migration from
BoxLib is available at Docs/Migration.
.. toctree::
:maxdepth: 1
:caption: Contents:
BuildingMFiX
TestingMFiX
Directory overview
------------------
.. table:: Overview of the MFiX-Exa directory structure
+------------+-----------------------------------------------------+
| File | Description |
+============+=====================================================+
| benchmarks | UC Benchmark cases (see benchmark/README.md) |
+------------+-----------------------------------------------------+
| src | Fortran source files |
+------------+-----------------------------------------------------+
| tests | Regression tests (see tests/README.md) |
+------------+-----------------------------------------------------+
| ThirdParty | External libraries sources |
+------------+-----------------------------------------------------+
| Tools | CMake configuration files |
+------------+-----------------------------------------------------+
Running the MFIX Test Suite
===========================
MFiX-Exa comes with several tests aimed at evaluating software functionalities.
The source files as well as the required input files for each test are located
in the ``tests`` directory. The ``tests`` directory is copied to the build
directory during MFIX-Exa configuration process. When a test is run (see
below), output files are stored in ``build_dir/tests/test-name``.
There are various dependencies for comparing test results.
* To compare results to archived flow slices stored in ``AUTOTEST``
directories with the case files, the environment variable ``FEXTRACT``
must point to the location of the AMReX `fextract` utility located
in the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally,
``numdiff`` must be installed for comparing results.
* To compare point-by-point field data, the environment variable
``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids``
found in the directory, ``amrex/Tools/PostProcessing/F_Src``.
Additionally, the environment variable ``MFIX_BENCHMARKS_HOME``
must point to the location of a local regression test data set.
See :ref:`sec:test:generate_data` for instructions on
creating a local regression test data set.
Run all tests
^^^^^^^^^^^^^
.. highlight:: bash
::
cd to mfix-build-dir
ctest
List all tests (without running them)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. highlight:: bash
::
cd to mfix-build-dir
ctest -N
Run a particular test by the index listed in ``ctest -N``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. highlight:: bash
::
cd to mfix-build-dir
ctest -I 3,3 # run the third test
Run a particular test by name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. highlight:: bash
::
cd to mfix-build-dir
ctest -R DEM01 # running all tests with "DEM01" in the test name
Run a particular test via make
------------------------------
.. highlight:: bash
::
cd to mfix-build-dir
make run_DEM01-x # running "DEM01-x" and output to the screen
Run specific
^^^^^^^^^^^^
If the environment variable GRID is defined, it specifies which grid types to
run for the test(s). If GRID variable is not defined, the default is to run
the tests for all grid types.
.. highlight:: bash
::
env GRID="tiled" ctest -R DEM01 # running all tests with "DEM01" for tiled grid
env GRID="single multiple" ctest -R DEM01 # running all tests with "DEM01" for single grid and multiple grid
ctest -R DEM01 # running all tests with "DEM01" for all grid types (single, multiple, tiled)
Run a user-defined case
^^^^^^^^^^^^^^^^^^^^^^^
.. highlight:: bash
::
./mfix inputs-myrun
``inputs-myrun`` is a text file containing the AMReX input parameters; this can
be named anything as long as it is the **first** argument following the
executable. Note that many of the problem parameters are still defined in
``mfix.dat``.
Regression testing
------------------
.. _sec:test:generate_data:
Generating local regression test data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Developers are encouraged to create local benchmark data for regression testing
prior to committing code the GitLab repository.
1. Create a location to store benchmark data and clone the MFiX and AMReX
repositories.
.. highlight:: bash
::
mkdir /home/user/exa-rt
mkdir /home/user/exa-rt/benchmark
cd /home/user/exa-rt
git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git
git clone https://github.com/AMReX-Codes/amrex.git
2. Create a local copy the regression test setup file from the MFiX repository.
.. highlight:: bash
::
cp mfix/RegressionTesting/MFIX-tests.ini MFIX-local.ini
3. Edit the local setup file. Specify the top level directories for test and
web output under the ``[main]`` heading.
.. highlight:: ini
::
[main]
testTopDir = /home/user/exa-rt/benchmark
webTopDir = /home/user/exa-rt/web
Specify the AMReX source directory location under the ``[AMReX]`` heading.
.. highlight:: ini
::
[AMReX]
dir = /home/user/exa-rt/amrex
branch = development
Specify the MFIX-Exa source directory location under the ``[source]`` heading.
.. highlight:: ini
::
[source]
dir = /home/user/exa-rt/mfix
branch = develop
4. Run the AMReX regression test tool. The second argument is a user supplied
comment.
.. highlight:: bash
::
cd /home/user/exa-rt
amrex/Tools/RegressionTesting/regtest.py --make_benchmarks "MFIX" MFIX-local.ini
Prerequisite: Environment Dependencies on Joule (Joule specific)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For the Joule environment, load the gnu module and set environment variables
first. If not on Joule, skip this step.
.. highlight:: bash
::
module load gnu/6.1.0
export CC=/nfs/apps/Compilers/GNU/6.1.0/bin/gcc
export CXX=/nfs/apps/Compilers/GNU/6.1.0/bin/g++
export F77=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran
export FC=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran
/* override table width restrictions */
.wy-table-responsive table td, .wy-table-responsive table th {
white-space: normal;
}
.wy-table-responsive {
margin-bottom: 24px;
max-width: 100%;
overflow: visible;
}
\ No newline at end of file
# -*- coding: utf-8 -*-
#
# amrex documentation build configuration file, created by
# sphinx-quickstart on Thu Oct 19 14:30:08 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
import sphinx_rtd_theme
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.mathjax',
'sphinx.ext.githubpages',
'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['ytemplates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'MFiX-Exa'
copyright = u'2017-2018, MFiX-Exa Team'
author = u'MFiX-Exa Team'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
#version =
# The full version, including alpha/beta/rc tags.
#release =
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# This is necessary to reference numbered figures
numfig = True
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_context = {
'css_files': [
'_static/theme_overrides.css', # overrides for wide tables in RTD theme
],
}
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# This is required for the alabaster theme
# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars
html_sidebars = {
'**': [
'about.html',
'navigation.html',
'relations.html', # needs 'show_related': True theme option to display
'searchbox.html',
'donate.html',
]
}
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'amrexdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'mfix.tex', u'MFiX-Exa Documentation',
u'MFiX-Exa Team', 'manual'),
]
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'mfix', u'MFiX-Exa Documentation',
[author], 1)
]
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'mfix', u'MFiX-Exa Documentation',
author, 'MFiX-Exa Term', "MFiX-Exa, it's MFiX... but now with more Exa!",
'Miscellaneous Aardvark'),
]
.. MFiX-Exa documentation master file, created by
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to MFiX-Exa's documentation
===================================
MFiX-Exa is a multi-phase flow simulation tool based on `MFiX-Classic
<http://mfix.netl.doe.gov>`_, incorporating the massively parallel,
block-structured adaptive mesh refinement (AMR) functionality of `AMReX
<http://amrex-codes.github.io>`_. MFiX-Exa is developed at LBNL and NETL, and
as part of the DOE's Exascale Computing Project.
The code for now lives at NETL; instructions for how to get access to the code
will be given below.
.. toctree::
:maxdepth: 1
:caption: Contents:
Introduction
See the User's Guide for more about MFiX-Exa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To build the User's Guide,
.. highlight:: bash
::
cd doc/UsersGuide
make
This will build a pdf of the MFiX-Exa User's Guide, which contains information
about the equations being solved, run-time parameters, checkpoint/restart
capability, options for visualization and more.
Notice
------
Neither the United States Government nor any agency thereof, nor any of their
employees, makes any warranty, expressed or implied, or assumes any legal
liability or responsibility for the accuracy, completeness, or usefulness of
any information, apparatus, product, or process disclosed or represents that
its use would not infringe privately owned rights.
* MFiX is provided without any user support for applications in the user's
immediate organization. It should not be redistributed in whole or in part.
* The use of MFIX is to be acknowledged in any published paper based on
computations using this software by citing the MFIX theory manual. Some of
the submodels are being developed by researchers outside of NETL. The use of
such submodels is to be acknowledged by citing the appropriate papers of the
developers of the submodels.
* The authors would appreciate receiving any reports of bugs or other
difficulties with the software, enhancements to the software, and accounts of
practical applications of this software.
Disclaimer
^^^^^^^^^^
This report was prepared as an account of work sponsored by an agency of the
United States Government. Neither the United States Government nor any agency
thereof, nor any of their employees, makes any warranty, express or implied, or
assumes any legal liability or responsibility for the accuracy, completeness,
or usefulness of any information, apparatus, product, or process disclosed, or
represents that its use would not infringe privately owned rights. Reference
herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply
its endorsement, recommendation, or favoring by the United States Government or
any agency thereof. The views and opinions of authors expressed herein do not
necessarily state or reflect those of the United States Government or any
agency thereof.
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`