Update code testing docs

.. _Chap:CITesting:

Continuous Integration

======================

The following regression tests are run every time a commit is pushed to the main

MFIX-Exa repository on the NETL gitlab.

For each of the tests in the chart below, there are

three directional variations; these are identified in the repository as, 

for example, FLD01-x, FLD01-y, and FLD01-z.  

For each direction, where appropriate, there are multiple versions, with the following notations:

  * SGS: single grid serial

  * MGS: multiple grid serial

  * TGS: tiled grid serial

  * MGP: multiple grid parallel

Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks.

All the FLD cases are fluid-only and steady state.

All the DEM cases are particle-only except for DEM06 and DEM07 which are fluid and particles;

these both use the "BVK2" drag type.

In all cases the particle data were read in from "particle_input.dat"

None of these tests have non-rectangular geometry.

"NSW" means "No Slip Wall" and "Per" is "periodic."

"MI/PO" refers to Mass     Inflow at the low end of the domain and Pressure Outflow at the high end.

"PI/PO" refers to Pressure Inflow at the low end of the domain and Pressure Outflow at the high end.

Additional detail about these problems is given in tests/README.md

Single-grid, single-process (SGS) particle-only tests:

+-------+----+----+----+------+------+-------+-----+--------------------+

| Test  | nx | ny | nz | bc_x | bc_y | bc_z  | Npa | Description        |

+=======+====+====+====+======+======+=======+=====+====================+

| DEM01 | 2  | 5  | 5  | NSW  | Per  | Per   | 1   | Freely falling     |

|       |    |    |    |      |      |       |     | particle with      |

|       |    |    |    |      |      |       |     | wall collision     |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM02 | 2  | 5  | 5  | NSW  | Per  | Per   | 1   | Multiple bounces   |

|       |    |    |    |      |      |       |     | with bounce height |

|       |    |    |    |      |      |       |     | measured           |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM03 | 2  | 5  | 5  | NSW  | Per  | Per   | 2   | Two stacked        |

|       |    |    |    |      |      |       |     | compressed         |

|       |    |    |    |      |      |       |     | particles          |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM04 | 4  | 4  | 4  | NSW  | Per  | Per   | 1   | Single particle    |

|       |    |    |    |      |      |       |     | slipping on a      |

|       |    |    |    |      |      |       |     | rough surface      |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM05 | 5  |  2 |  5 | Per  | Per  | Per   | 93  | Oblique particle   |

|       |    |    |    |      |      |       |     | collisions         |

|       |    |    |    |      |      |       |     |                    |

+-------+----+----+----+------+------+-------+-----+--------------------+

Steady-state fluid-only tests:

+-------+-----+----+----+----+-------+------+------+-----+----------------------+

| Test  |     | nx | ny | nz | bc_x  | bc_y | bc_z | Ng  | Np                   |

+=======+=====+====+====+====+=======+======+======+=====+======================+

| FLD01 |     | 8  | 8  | 4  | Per   | NSW  | Per  | Poiseuille flow            |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 4   | 1  |                 |

|       | MGP |    |    |    |       |      |      | 4   | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

| FLD02 |     | 80 |16  | 16 | MI/PO | NSW  | NSW  | Couette flow               |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 40  | 1  |                 | 

|       | MGP |    |    |    |       |      |      | 40  | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

| FLD03 |     | 8  | 8  | 4  | PI/PO | NSW  | Per  | Poiseuille flow            |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 4   | 1  |                 |

|       | MGP |    |    |    |       |      |      | 4   | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

Coupled particle/fluid tests:

+-------+-----+----+----+----+------+------+------+------+----+--------------------+

| Test  |     | nx | ny | nz | bc_x | bc_y | bc_z | Npa  | Ng | Np                 |

+=======+=====+====+====+====+======+======+======+======+====+====================+

| DEM06 |     | 50 | 5  | 5  |  NSW | NSW  | NSW  | 1    | Single particle falling | 

|       |     |    |    |    |      |      |      |      | under gravity           |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

|       | SGS |    |    |    |      |      |      |      | 1  | 1  |               |

|       | MGS |    |    |    |      |      |      |      | 10 | 1  |               |

|       | MGP |    |    |    |      |      |      |      | 10 | 8  |               |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

| DEM07 |     | 20 | 20 | 20 | Per  | Per  | Per  | 1222 | Homogeneous cooling     |

|       |     |    |    |    |      |      |      |      | system                  |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

|       | SGS |    |    |    |      |      |      |      | 1  | 1  |               |

|       | MGS |    |    |    |      |      |      |      | 8  | 1  |               |

|       | MGP |    |    |    |      |      |      |      | 8  | 8  |               |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

Continuous Integration (CI)

===========================

MFIX-Exa uses a GitLab-based Continuous Integration (CI) pipeline to ensure

code quality, portability, and correctness across multiple compilers, GPU

backends, and build configurations. This page describes what the CI runs,

how to reproduce CI behavior locally, and provides a high-level overview of

the MFIX-Exa test suite.

When the CI Pipeline Runs

-------------------------

The CI pipeline is triggered under the following conditions:

* **Merge requests** – runs linting, warnings builds, and multi-backend builds.

* **Commits to the ``develop`` branch** – runs the full test pipeline.

* **Tag pushes** – used for releases.

* **Manual runs** – triggered from the GitLab web interface.

CI Pipeline Structure

---------------------

The MFIX-Exa CI is organized into two distinct execution paths:

* **Merge Request (MR) pipelines** – run on every push to a merge request.

  These pipelines focus on code quality, portability, and build correctness.

  They include the **Lint**, **Warnings**, and **Build** stages.

* **Post-merge pipelines** – run only after changes are merged into the

  ``develop`` branch. These pipelines run the full **CTest-based test suite**

  and the specialized **Ascent** and **MPMD** tests.

This separation ensures fast feedback during code review while reserving

the more expensive tests for post-merge validation.

Merge Request Pipeline Stages

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Lint**

  Static analysis and formatting checks:

  * Shell scripts (``shellcheck``)

  * Spelling (``codespell``)

  * Tabs and trailing whitespace removal

  * Python linting (``pylint``) for the GUI

**Warnings Builds**

  Compiler-strict builds using GCC and Clang in both Debug and Release modes.

  These builds enable extensive warning flags (``-Wall``, ``-Wextra``,

  ``-Werror``, etc.) to catch portability and correctness issues early.

**Builds**

  Multiple configurations are compiled to ensure portability across compilers

  and GPU backends:

  * CPU-only builds

  * GPU-enabled builds (CUDA, HIP, SYCL)

  * CSG-enabled builds

  * MPMD-enabled builds

  * Unit tests

These MR-stage jobs verify that MFIX-Exa builds cleanly and consistently

before code is merged.

Post-Merge Test Pipeline

~~~~~~~~~~~~~~~~~~~~~~~~

After changes are merged into the ``develop`` branch, the CI runs the full

MFIX-Exa test suite. These jobs are triggered by ``rules:`` blocks that

restrict execution to the ``develop`` branch.

The post-merge pipeline includes:

**CTest-based MFIX-Exa Test Suite**

  Tests are grouped by label:

  * ``SGS`` – single-grid serial tests

  * ``MGS`` – multi-grid serial tests

  * ``TGS`` – tiled-grid serial tests

  * ``MGP`` – multi-grid parallel tests

  * ``TGPO`` – tiled-grid parallel OpenMP tests

  * ``TGSO`` – tiled-grid serial OpenMP tests

**Ascent Visualization Test**

  Runs a small HCS benchmark with Ascent enabled and verifies image output.

**MPMD Tutorial Test**

  Runs the MPMD tutorial example and checks numerical output and generated

  images.

How to Reproduce CI Locally

---------------------------

Building MFIX-Exa

~~~~~~~~~~~~~~~~~

A typical local build mirrors the CI configuration. For example:

.. code-block:: bash

   cmake -S <path-to-mfix> -B build \

         -DCMAKE_BUILD_TYPE=Release \

         -G Ninja

   cmake --build build --target build_tests unit_tests

Additional options (e.g., enabling MPI, OpenMP, GPU backends, or CSG)

can be added as needed.

Running the MFIX-Exa Test Suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CTest is used to run the MFIX-Exa test suite. All tests are available

locally, regardless of which subset the CI runs.

List all tests:

.. code-block:: bash

   ctest --test-dir build/ -N

Run all tests:

.. code-block:: bash

   ctest --test-dir build/

Run a specific test by name:

.. code-block:: bash

   ctest --test-dir build/ -R DEM01

Run a specific test by index:

.. code-block:: bash

   ctest --test-dir build/ -I 3,3

Run selected the signle-grid serial DEM tests in the ``y`` direction:

.. code-block:: bash

   ctest --test-dir build/ -R "DEM(01|03|04|05|06)-y_SGS|DEM07_SGS"

Running Unit Tests

~~~~~~~~~~~~~~~~~~

.. code-block:: bash

   cmake --build build --target unit_tests

   ctest --test-dir build/ -R unit_test -V

Summary of MFIX-Exa Test Categories

-----------------------------------

The MFIX-Exa test suite includes fluid-only tests, particle-only tests,

and coupled fluid/particle tests. The CI runs a representative subset of

these, while the full suite is available locally.

Below is a high-level summary of the test categories and representative

cases.

.. list-table::

   :header-rows: 1

   :widths: 15 20 65

   * - Test

     - Type

     - Description

   * - ``FLD01``

     - Fluid

     - 2D channel (Poiseuille) flow in a periodic domain.

   * - ``FLD02``

     - Fluid

     - 2D Couette flow in a channel.

   * - ``FLD03``

     - Fluid

     - Pressure-driven channel flow with specified pressure boundaries.

   * - ``DEM01``

     - DEM

     - Freely falling particle with wall collision; verifies spring-dashpot model.

   * - ``DEM02``

     - DEM

     - Repeated particle bouncing; compares rebound height to theory.

   * - ``DEM03``

     - DEM

     - Two compressed particles between walls; tests multi-particle contact.

   * - ``DEM04``

     - DEM

     - Rolling/slipping particle on a rough surface; tests friction model.

   * - ``DEM05``

     - DEM

     - Oblique particle collisions with a flat wall; tests tangential restitution.

   * - ``DEM06``

     - Coupled

     - Single particle terminal velocity in a fluid; tests drag models.

   * - ``DEM07``

     - Coupled

     - Homogeneous Cooling System (HCS); tests granular temperature decay.

Details for each case are available in ``tests/README.md``.

Relationship to Regression Tests

--------------------------------

The CI pipeline **does not** run the CCSE regression suite.

* The **Regression Tests** page documents the CCSE/AMReX regression framework.

* The CI page documents the **CTest-based MFIX-Exa Test Suite** and infrastructure tests.

These two systems are complementary:

* CI ensures build correctness and basic functional behavior.

* Regression tests ensure numerical consistency across platforms and versions.

Developer Expectations

----------------------

* All CI jobs must pass before merging into ``develop``.

* Lint and warnings jobs help maintain code quality.

* Build jobs ensure portability across compilers and GPU backends.

* Test jobs ensure functional correctness.

* CI artifacts contain detailed logs for debugging failures.

.. _Chap:NightlyTesting :

Nightly Tests

=============

The following regression tests are run nightly with MFIX-Exa.   The plotfiles generated in each night's test 

are compared with the benchmark plotfiles using the AMReX :cpp:`fcompare` utility to compare the mesh data

and :cpp:`particle_compare` to compare the particle data.

The results of these tests can be found at https://ccse.lbl.gov/pub/RegressionTesting/MFIX-Exa/

Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks.

"Auto" means the particles were generated automatically with the random number

generator; if "Auto" is not specified the particle data were read in from "particle_input.dat"

These first tests have both fluid and particles and are run in rectangular geometries;

all tests except DEM06 use drag type "BVK2". 

"NSW" means "No Slip Wall" and "Per" is "periodic."

"MI/PO" refers to Mass Inflow at the low end of the domain and Pressure Outflow at the high end.

+-------------------+----+--------+------+-------+----+----+----------------------+

| Test              | nx | bc_x   | EB   | Npa   | Ng | Np | What does this test? |

|                   | ny | bc_y   |      |       |    |    |                      |

|                   | nz | bc_z   |      |       |    |    |                      |

+===================+====+========+======+=======+====+====+======================+

| BENCH01           | 32 | Per    | None | 5005  | 1  | 1  | Triply periodic      |

| Size0001          | 32 | Per    |      |       |    |    |                      |

|                   | 32 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH01           | 64 | Per    | None | 40040 | 8  | 4  | Replicate            |

| Size0001          | 64 | Per    |      |       |    |    |                      |

| replicate         | 64 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH01           | 32 | Per    | None | 5005  | 8  | 4  | Restart              |

| Size0001          | 32 | Per    |      |       |    |    |                      |

| restart           | 32 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH02           | 10 | Per    | None | 1611  | 1  | 1  | Mixed NSW / Per      |

| Size0001          | 10 | NSW    |      |       |    |    |                      |

|                   | 10 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH02           | 10 | NSW    | None | 1611  | 1  | 1  | NSW on all faces     |

| Size0001          | 10 | NSW    |      |       |    |    |                      |

| walls             | 10 | NSW    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH03           | 4  | Per    | None | 2500  | 1  | 1  | Mixed MI/PO + Per    |

| Size0001          | 50 | MI/PO  |      |       |    |    |                      |

|                   | 4  | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH04           | 4  | Per    | None | 224   | 1  | 1  | Triply periodic      |

| Size0001          | 50 | Per    |      |       |    |    |                      |

|                   | 4  | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| DEM06             | 5  | Per    | None | 1     | 10 | 4  | Single particle      |

| z multiple        | 5  | Per    |      |       |    |    | falling in fluid     |

|                   | 50 | MI/PO  |      |       |    |    | (user_drag)          |

+-------------------+----+--------+------+-------+----+----+----------------------+

This second set of tests have both fluid and particles and are run in cylindrial geometries

interior to the domain boundaries; they also use drag type "BVK2".  Here "IGN" means

those domain boundaries should be ignored because they are outside the EB boundary.

+-------------------+----+-------+------+--------+----+----+----------------------+

| Test              | nx | bc_x  | EB   | Npa    | Ng | Np | What does this test? |

|                   | ny | bc_y  |      |        |    |    |                      |

|                   | nz | bc_z  |      |        |    |    |                      |

+===================+====+=======+======+========+====+====+======================+

| BENCH05           | 40 | MI/PO | Cyl  | 7949   | 4  | 4  | EB in parallel       |

| Size0008          | 10 | IGN   |      | Auto   |    |    |                      |

|                   | 10 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 7968   | 4  | 1  | EB in serial         |

| Size0008          | 10 | IGN   |      | Auto   |    |    |                      |

| serial            | 10 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 36672  | 16 | 4  | Regrid & dual grid   |

| Size0008          | 20 | IGN   |      | Auto   |    |    |                      |

| medium            | 20 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 157106 | 16 | 4  | Regrid & dual grid   |

| Size0008          | 40 | IGN   |      | Auto   |    |    |                      |

| wide              | 40 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH06           | 40 | Per   | Cyl  | 627    | 4  | 1  | EB                   |

| Size0008          | 10 | IGN   |      | Auto   |    |    | with periodic        |

| serial            | 10 | IGN   |      |        |    |    | serial               |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH06           | 40 | Per   | Cyl  | 624    | 4  | 4  | EB                   |

| Size0008          | 10 | IGN   |      | Auto   |    |    | with periodic        |

|                   | 10 | IGN   |      |        |    |    | parallel             |

+-------------------+----+-------+------+--------+----+----+----------------------+

This third set of tests is particles-only in rectangular geometries.

+-------------------+----+-------+------+--------+----+----+----------------------+

| Test              | nx | bc_x  | EB   | Npa    | Ng | Np | What does this test? |

|                   | ny | bc_y  |      |        |    |    |                      |

|                   | nz | bc_z  |      |        |    |    |                      |

+===================+====+=======+======+========+====+====+======================+

| DEM01             | 4  | NSW   | None |   1    | 1  | 1  | Particle  only       |

| x single          | 4  | NSW   |      |        |    |    |                      |

|                   | 4  | NSW   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| DEM03             | 5  | Per   | None |   2    | 1  | 1  | Particles only       |

| z single          | 5  | Per   |      |        |    |    |                      |

|                   | 2  | NSW   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| DEM04             | 4  | NSW   | None |   1    | 1  | 1  | Particles only       |

| z single          | 4  | Per   |      |        |    |    |                      |

|                   | 4  | Per   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

Regression testing

.. _Chap:RegressionTesting:

Regression Testing

==================

Generating local regression test data

-------------------------------------

The MFIX-Exa regression test suite is built on top of the AMReX

regression testing framework, which is developed and maintained by the

AMReX team. The framework provides tools for building applications,

running benchmark problems, and comparing results against reference

datasets. The AMReX regression testing system is open source and

available at:

Developers are encouraged to create local benchmark data for regression

testing prior to committing code the GitLab repository.

   https://github.com/AMReX-Codes/regression_testing

1. Create a location to store benchmark data and clone the MFIX-Exa and

   AMReX repositories.

   .. code:: shell

This page describes how to configure and run the regression tests, how

to customize the test configuration files, and provides a summary of the

available benchmark problems.

       mkdir /home/user/exa-rt

       mkdir /home/user/exa-rt/benchmark

       cd /home/user/exa-rt

       git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git

       git clone https://github.com/AMReX-Codes/amrex.git

2. Create a local copy the regression test setup file from the MFIX-Exa

   repository.

Overview

--------

   .. code:: shell

Regression tests are used to ensure that changes to MFIX-Exa do not

introduce unintended differences in numerical results. These tests:

       cp mfix/RegressionTesting/MFiX-tests.ini MFiX-local.ini

* build MFIX-Exa and AMReX in a controlled configuration,

* run a suite of benchmark problems,

* compare the results against reference data,

* report any differences.

3. Edit the local setup file. Specify the top level directories for test

   and web output under the ``[main]`` heading.

The regression suite is **not** part of the CI pipeline. CI runs the

CTest-based MFIX-Exa Test Suite, while regression tests must be run

manually by developers.

   .. code:: shell

       [main]

       testTopDir = /home/user/exa-rt/benchmark

       webTopDir =  /home/user/exa-rt/web

Regression Test Configuration Files

-----------------------------------

   Specify the AMReX source directory location under the ``[AMReX]``

   heading.

MFIX-Exa provides **two separate regression test configuration

files**, located in the ``tests/`` directory:

   .. code:: shell

.. code-block:: text

       [AMReX]

       dir = /home/user/exa-rt/amrex

       branch = development

   tests/MFIX-tests-CPU.ini

   tests/MFIX-tests-GPU.ini

Use the CPU configuration file for CPU-only regression testing and the GPU

configuration file for CUDA-enabled regression testing. At this time,

MFIX-Exa does not test configurations for HIP (AMD) or SYCL (Intel) backends.

For GPU (CUDA) regression tests, the benchmark datasets are generated using a

CPU-only build by setting ``-DAMReX_GPU_BACKEND=NONE`` and

``-DMFIX_GPU_BACKEND=NONE``. This ensures that CPU and GPU runs are

compared against a common reference dataset. Because GPU runs may exhibit

small numerical differences relative to CPU-generated benchmarks, the

regression tolerances for GPU tests are typically less strict.

Structure of the Configuration Files

------------------------------------

Each ``.ini`` file contains several sections:

* ``[main]`` – global settings for the regression framework

* ``[AMReX]`` – AMReX source location and CMake options

* ``[source]`` – MFIX-Exa source location and CMake options

* ``[BENCHXX-*]`` – individual benchmark problem definitions

   Specify the MFIX-Exa source directory location under the ``[source]``

   heading.

Only a small number of entries typically need to be modified by users.

   .. code:: shell

User-Editable Fields

--------------------

Required Edits

~~~~~~~~~~~~~~

Users must update the following fields in the ``ini`` file.

``MFIX-tests-GPU.ini``:

* ``testTopDir`` – directory where regression outputs will be written.

* ``webTopDir`` – directory for HTML reports (optional).

* ``[AMReX]/dir`` – path to the AMReX source tree, typically the version

  located in the ``subprojects/amrex`` directory.

* ``[source]/dir`` – path to the MFIX-Exa source tree.

Example:

.. code-block:: ini

   testTopDir = /path/to/regression/output

   webTopDir  = /path/to/regression/web

   [AMReX]

   dir = /path/to/mfix-exa/subprojects/amrex

   [source]

       dir = /home/user/exa-rt/mfix

       branch = develop

   dir = /path/to/mfix-exa

When maintaining both CPU and GPU regression suites on the same system,

the ``testTopDir`` and ``webTopDir`` entries in each ``ini`` file should

point to different directories. This ensures that the benchmark datasets

and test results for CPU and GPU runs remain separate. Mixing these

directories can lead to incorrect comparisons, overwritten benchmark

files, or misleading regression reports.

Optional Edits

~~~~~~~~~~~~~~

These entries may be modified depending on the system:

* ``MPIcommand`` – command used to launch MPI jobs

* ``MPIhost`` – hostname for MPI execution (optional)

* ``numMakeJobs`` – number of parallel build jobs

* ``branch`` – AMReX or MFIX-Exa branch to check out

Advanced Edits

~~~~~~~~~~~~~~

These entries are rarely modified:

* ``cmakeSetupOpts`` in ``[AMReX]`` or ``[source]``

* ``runtime_params`` inside individual benchmark sections

* ``numprocs`` or ``numthreads`` for scaling studies

Running the Regression Tests

----------------------------

The MFIX-Exa regression suite is driven by the AMReX regression testing

framework. Before creating or running any regression tests, ensure that

the directories specified in the ``ini`` configuration file exist:

* ``testTopDir`` – directory where benchmark and test output will be written

* ``webTopDir`` – directory where HTML reports will be generated

Both directories must be created manually before running the regression

framework.

In the examples below, ``regression_testing`` refers to the AMReX

regression testing repository available at:

   https://github.com/AMReX-Codes/regression_testing

Creating Benchmark Datasets

~~~~~~~~~~~~~~~~~~~~~~~~~~~

Benchmark datasets serve as the reference results against which future

runs are compared. They must be created before running the regression

suite for the first time, or whenever intentional changes to MFIX-Exa

require updated reference results.

Create benchmark datasets for **all** tests listed in the ``ini`` file:

.. code-block:: bash

   ./regression_testing/regtest.py \

       --make_benchmarks "creating initial benchmark datasets" \

       MFIX-tests.ini

Create benchmark datasets for a **subset** of tests:

.. code-block:: bash

   ./regression_testing/regtest.py \

       --make_benchmarks "creating initial benchmark datasets" \

       --tests 'LIST OF BENCHMARK CASES' \

       MFIX-tests.ini

Running the Regression Tests

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To run the regression suite without updating AMReX or MFIX-Exa, use the

``--no_update all`` flag. This prevents the framework from checking out

the branches listed in the ``ini`` file and is recommended when testing

local modifications.

Run specific tests with a descriptive note:

.. code-block:: bash

   ./regression_testing/regtest.py \

       --no_update all \

       --note "test after making change XYZ" \

       --tests 'BENCH01-Size0001 BENCH02-Size0001' \

       MFIX-tests.ini

Updating Benchmark Results

~~~~~~~~~~~~~~~~~~~~~~~~~~

If a code modification is known to change numerical results, the

benchmark datasets must be updated. There are two ways to do this:

1. **Recreate benchmarks using ``--make_benchmarks``**

   (recommended when changes are intentional and validated)

2. **Copy benchmark results from a failed test run**

   (useful when the regression suite has already produced new output)

Update benchmarks using the ``copy_benchmarks`` option:

.. code-block:: bash

   ./regression_testing/regtest.py \

       --no_update all \

       --copy_benchmarks "Updating after submodules" \

       MFIX-tests.ini

This replaces the existing benchmark plotfiles with those generated

during the most recent test run.

Interpreting Results

--------------------

After the tests complete, the framework generates:

* a summary of passed/failed tests,

* detailed logs for each benchmark,

* optional HTML reports (if ``webTopDir`` is set),

* particle comparison results (if ``compareParticles = 1``).

A test is considered failed if:

* MFIX-Exa output differs from the benchmark data,

* the simulation crashes or produces invalid output,

* particle trajectories differ beyond tolerance.

Benchmark Summary

-----------------

The MFIX-Exa regression suite includes a broad set of benchmark problems

covering fluid flow, particle dynamics, and coupled multiphase physics.

A high-level summary is provided below.

.. list-table::

   :header-rows: 1

   :widths: 15 20 65

   * - Benchmark

     - Category

     - Description

   * - ``BENCH01``

     - HCS

     - Homogeneous Cooling System (HCS) with multiple advection and redistribution options; includes restart and replicate tests.

   * - ``BENCH02``

     - Settling

     - Single-particle settling in fluid; includes no-fluid and wall-bounded variants.

   * - ``BENCH03``

     - Fluidized bed

     - Small fluidized bed with gas–solid interaction.

   * - ``BENCH04``

     - Square riser

     - Gas–solid riser flow in a square duct.

   * - ``BENCH05``

     - Cylindrical fluidized bed

     - Multiple grid sizes and physics options; includes chemistry and covered-grid tests.

   * - ``BENCH06``

     - Cylindrical riser

     - Gas–solid riser with multiple MPI configurations.

   * - ``BENCH08``

     - PIC

     - Particle-in-cell tests including restart variants.

   * - ``BENCH09``

     - Refinement

     - AMR refinement test for particle and fluid fields.

   * - ``DEM01–DEM06``

     - DEM

     - Particle-only verification tests (free fall, bouncing, compression, rolling, oblique collisions, terminal velocity).

   * - ``TracerAdv``

     - Tracer transport

     - Tracer advection with optional Godunov/MOL advection schemes.

Relationship to CI Tests

------------------------

4. Run the AMReX regression test tool. The second argument is a user

   supplied comment.

The regression suite is **separate** from the MFIX-Exa CI pipeline:

   .. code:: shell

* CI runs the **CTest-based MFIX-Exa Test Suite**.

* Regression tests use the **AMReX regression framework**.

* CI tests focus on build correctness and basic functionality.

* Regression tests focus on numerical correctness and reproducibility.

       cd /home/user/exa-rt

       amrex/Tools/RegressionTesting/regtest.py --make_benchmarks "MFiX" MFiX-local.ini

Both systems complement each other and should be used regularly during

development.

+------------------------------------------------------------------------------------------------------------------------+

| ## Prerequisite: Environment Dependencies on Joule (Joule specific)                                                    |

+------------------------------------------------------------------------------------------------------------------------+

| For the Joule environment, load the gnu module and set environment variables first. If not on Joule, skip this step.   |

+------------------------------------------------------------------------------------------------------------------------+

 .. code:: shell

Developer Expectations

----------------------

    module load gnu/6.1.0

    export CC=/nfs/apps/Compilers/GNU/6.1.0/bin/gcc 

    export CXX=/nfs/apps/Compilers/GNU/6.1.0/bin/g++

    export F77=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran

    export FC=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran

* CPU and GPU regression tests should be run when modifying multiphase

  physics, AMReX interfaces, or GPU kernels.

* Any regression failure must be investigated and resolved before merging.

* Benchmark data should only be updated when numerical changes are

  intentional and documented.

Running the MFIX-Exa Test Suite

===============================

MFIX-Exa comes with several tests aimed at evaluating software

functionalities. The source files as well as the required input files

for each test are located in the ``tests`` directory. The ``tests``

directory is copied to the build directory during MFIX-Exa configuration

process. When a test is run (see below), output files are stored in

``build_dir/tests/test-name``.

There are various dependencies for comparing test results.

o To compare results to archived flow slices stored in ``AUTOTEST``

directories with the case files, the environment variable ``FEXTRACT``

must point to the location of the AMReX ``fextract`` utility located in

the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally,

``numdiff`` must be installed for comparing results.

o To compare point-by-point field data, the environment variable

``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids``

found in the directory, ``amrex/Tools/PostProcessing/F_Src``.

Additionally, the environment variable ``MFIX_BENCHMARKS_HOME`` must

point to the location of a local regression test data set. See

*Generating local regression test data* for instructions on creating a

local regression test data set.

Run all tests

-------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest

List all tests (without running them)

-------------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -N

Run a particular test by the index listed in ctest -N

-----------------------------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -I 3,3             # run the third test

Run a particular test by name

-----------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -R DEM01  # running all tests with "DEM01" in the test name

Run a particular test via make

------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > make run_DEM01-x  # running "DEM01-x" and output to the screen

Run specific

------------

If the environment variable GRID is defined, it specifies which grid

types to run for the test(s). If GRID variable is not defined, the

default is to run the tests for all grid types. > env GRID="tiled" ctest

-R DEM01 # running all tests with "DEM01" for tiled grid > env

GRID="single multiple" ctest -R DEM01 # running all tests with "DEM01"

for single grid and multiple grid > ctest -R DEM01 # running all tests

with "DEM01" for all grid types (single, multiple, tiled)

Run a user-defined case

-----------------------

.. code:: shell

    > ./mfix inputs-myrun

*inputs-myrun* is a text file containing the AMReX input parameters;

this can be named anything as long as it is the **first** argument

following the executable. Note that many of the problem parameters are

still defined in *mfix.dat*.

   :maxdepth: 0

   :caption: Tests and Benchmarks:

   test_benchmarks/RunningTestSuite

   test_benchmarks/RegressionTesting

   test_benchmarks/CITests

   test_benchmarks/NightlyTests

   test_benchmarks/RegressionTesting

   test_benchmarks/qualitative_bencharks/index

   test_benchmarks/verification/index