Update code testing docs (!158)

   :maxdepth: 0

   :caption: Tests and Benchmarks:

   test_benchmarks/RunningTestSuite

   test_benchmarks/RegressionTesting

   test_benchmarks/CITests

   test_benchmarks/NightlyTests

   test_benchmarks/RegressionTesting

   test_benchmarks/qualitative_bencharks/index

   test_benchmarks/verification/index

.. _Chap:CITesting:

Continuous Integration

======================

The following regression tests are run every time a commit is pushed to the main

MFIX-Exa repository on the NETL gitlab.

For each of the tests in the chart below, there are

three directional variations; these are identified in the repository as, 

for example, FLD01-x, FLD01-y, and FLD01-z.  

For each direction, where appropriate, there are multiple versions, with the following notations:

  * SGS: single grid serial

  * MGS: multiple grid serial

  * TGS: tiled grid serial

  * MGP: multiple grid parallel

Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks.

All the FLD cases are fluid-only and steady state.

All the DEM cases are particle-only except for DEM06 and DEM07 which are fluid and particles;

these both use the "BVK2" drag type.

In all cases the particle data were read in from "particle_input.dat"

None of these tests have non-rectangular geometry.

"NSW" means "No Slip Wall" and "Per" is "periodic."

"MI/PO" refers to Mass     Inflow at the low end of the domain and Pressure Outflow at the high end.

"PI/PO" refers to Pressure Inflow at the low end of the domain and Pressure Outflow at the high end.

Additional detail about these problems is given in tests/README.md

Single-grid, single-process (SGS) particle-only tests:

+-------+----+----+----+------+------+-------+-----+--------------------+

| Test  | nx | ny | nz | bc_x | bc_y | bc_z  | Npa | Description        |

+=======+====+====+====+======+======+=======+=====+====================+

| DEM01 | 2  | 5  | 5  | NSW  | Per  | Per   | 1   | Freely falling     |

|       |    |    |    |      |      |       |     | particle with      |

|       |    |    |    |      |      |       |     | wall collision     |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM02 | 2  | 5  | 5  | NSW  | Per  | Per   | 1   | Multiple bounces   |

|       |    |    |    |      |      |       |     | with bounce height |

|       |    |    |    |      |      |       |     | measured           |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM03 | 2  | 5  | 5  | NSW  | Per  | Per   | 2   | Two stacked        |

|       |    |    |    |      |      |       |     | compressed         |

|       |    |    |    |      |      |       |     | particles          |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM04 | 4  | 4  | 4  | NSW  | Per  | Per   | 1   | Single particle    |

|       |    |    |    |      |      |       |     | slipping on a      |

|       |    |    |    |      |      |       |     | rough surface      |

+-------+----+----+----+------+------+-------+-----+--------------------+

| DEM05 | 5  |  2 |  5 | Per  | Per  | Per   | 93  | Oblique particle   |

|       |    |    |    |      |      |       |     | collisions         |

|       |    |    |    |      |      |       |     |                    |

+-------+----+----+----+------+------+-------+-----+--------------------+

Steady-state fluid-only tests:

+-------+-----+----+----+----+-------+------+------+-----+----------------------+

| Test  |     | nx | ny | nz | bc_x  | bc_y | bc_z | Ng  | Np                   |

+=======+=====+====+====+====+=======+======+======+=====+======================+

| FLD01 |     | 8  | 8  | 4  | Per   | NSW  | Per  | Poiseuille flow            |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 4   | 1  |                 |

|       | MGP |    |    |    |       |      |      | 4   | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

| FLD02 |     | 80 |16  | 16 | MI/PO | NSW  | NSW  | Couette flow               |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 40  | 1  |                 | 

|       | MGP |    |    |    |       |      |      | 40  | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

| FLD03 |     | 8  | 8  | 4  | PI/PO | NSW  | Per  | Poiseuille flow            |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

|       | SGS |    |    |    |       |      |      | 1   | 1  |                 |

|       | MGS |    |    |    |       |      |      | 4   | 1  |                 |

|       | MGP |    |    |    |       |      |      | 4   | 8  |                 |

+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+

Coupled particle/fluid tests:

+-------+-----+----+----+----+------+------+------+------+----+--------------------+

| Test  |     | nx | ny | nz | bc_x | bc_y | bc_z | Npa  | Ng | Np                 |

+=======+=====+====+====+====+======+======+======+======+====+====================+

| DEM06 |     | 50 | 5  | 5  |  NSW | NSW  | NSW  | 1    | Single particle falling | 

|       |     |    |    |    |      |      |      |      | under gravity           |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

|       | SGS |    |    |    |      |      |      |      | 1  | 1  |               |

|       | MGS |    |    |    |      |      |      |      | 10 | 1  |               |

|       | MGP |    |    |    |      |      |      |      | 10 | 8  |               |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

| DEM07 |     | 20 | 20 | 20 | Per  | Per  | Per  | 1222 | Homogeneous cooling     |

|       |     |    |    |    |      |      |      |      | system                  |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

|       | SGS |    |    |    |      |      |      |      | 1  | 1  |               |

|       | MGS |    |    |    |      |      |      |      | 8  | 1  |               |

|       | MGP |    |    |    |      |      |      |      | 8  | 8  |               |

+-------+-----+----+----+----+------+------+------+------+----+----+---------------+

Continuous Integration (CI)

===========================

MFIX-Exa uses a GitLab-based Continuous Integration (CI) pipeline to ensure

code quality, portability, and correctness across multiple compilers, GPU

backends, and build configurations. This page describes what the CI runs,

how to reproduce CI behavior locally, and provides a high-level overview of

the MFIX-Exa test suite.

When the CI Pipeline Runs

-------------------------

The CI pipeline is triggered under the following conditions:

* **Merge requests** – runs linting, warnings builds, and multi-backend builds.

* **Commits to the ``develop`` branch** – runs the full test pipeline.

* **Tag pushes** – used for releases.

* **Manual runs** – triggered from the GitLab web interface.

CI Pipeline Structure

---------------------

The MFIX-Exa CI is organized into two distinct execution paths:

* **Merge Request (MR) pipelines** – run on every push to a merge request.

  These pipelines focus on code quality, portability, and build correctness.

  They include the **Lint**, **Warnings**, and **Build** stages.

* **Post-merge pipelines** – run only after changes are merged into the

  ``develop`` branch. These pipelines run the full **CTest-based test suite**

  and the specialized **Ascent** and **MPMD** tests.

This separation ensures fast feedback during code review while reserving

the more expensive tests for post-merge validation.

Merge Request Pipeline Stages

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Lint**

  Static analysis and formatting checks:

  * Shell scripts (``shellcheck``)

  * Spelling (``codespell``)

  * Tabs and trailing whitespace removal

  * Python linting (``pylint``) for the GUI

**Warnings Builds**

  Compiler-strict builds using GCC and Clang in both Debug and Release modes.

  These builds enable extensive warning flags (``-Wall``, ``-Wextra``,

  ``-Werror``, etc.) to catch portability and correctness issues early.

**Builds**

  Multiple configurations are compiled to ensure portability across compilers

  and GPU backends:

  * CPU-only builds

  * GPU-enabled builds (CUDA, HIP, SYCL)

  * CSG-enabled builds

  * MPMD-enabled builds

  * Unit tests

These MR-stage jobs verify that MFIX-Exa builds cleanly and consistently

before code is merged.

Post-Merge Test Pipeline

~~~~~~~~~~~~~~~~~~~~~~~~

After changes are merged into the ``develop`` branch, the CI runs the full

MFIX-Exa test suite. These jobs are triggered by ``rules:`` blocks that

restrict execution to the ``develop`` branch.

The post-merge pipeline includes:

**CTest-based MFIX-Exa Test Suite**

  Tests are grouped by label:

  * ``SGS`` – single-grid serial tests

  * ``MGS`` – multi-grid serial tests

  * ``TGS`` – tiled-grid serial tests

  * ``MGP`` – multi-grid parallel tests

  * ``TGPO`` – tiled-grid parallel OpenMP tests

  * ``TGSO`` – tiled-grid serial OpenMP tests

**Ascent Visualization Test**

  Runs a small HCS benchmark with Ascent enabled and verifies image output.

**MPMD Tutorial Test**

  Runs the MPMD tutorial example and checks numerical output and generated

  images.

How to Reproduce CI Locally

---------------------------

Building MFIX-Exa

~~~~~~~~~~~~~~~~~

A typical local build mirrors the CI configuration. For example:

.. code-block:: bash

   cmake -S <path-to-mfix> -B build \

         -DCMAKE_BUILD_TYPE=Release \

         -G Ninja

   cmake --build build --target build_tests unit_tests

Additional options (e.g., enabling MPI, OpenMP, GPU backends, or CSG)

can be added as needed.

Running the MFIX-Exa Test Suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CTest is used to run the MFIX-Exa test suite. All tests are available

locally, regardless of which subset the CI runs.

List all tests:

.. code-block:: bash

   ctest --test-dir build/ -N

Run all tests:

.. code-block:: bash

   ctest --test-dir build/

Run a specific test by name:

.. code-block:: bash

   ctest --test-dir build/ -R DEM01

Run a specific test by index:

.. code-block:: bash

   ctest --test-dir build/ -I 3,3

Run selected the signle-grid serial DEM tests in the ``y`` direction:

.. code-block:: bash

   ctest --test-dir build/ -R "DEM(01|03|04|05|06)-y_SGS|DEM07_SGS"

Running Unit Tests

~~~~~~~~~~~~~~~~~~

.. code-block:: bash

   cmake --build build --target unit_tests

   ctest --test-dir build/ -R unit_test -V

Summary of MFIX-Exa Test Categories

-----------------------------------

The MFIX-Exa test suite includes fluid-only tests, particle-only tests,

and coupled fluid/particle tests. The CI runs a representative subset of

these, while the full suite is available locally.

Below is a high-level summary of the test categories and representative

cases.

.. list-table::

   :header-rows: 1

   :widths: 15 20 65

   * - Test

     - Type

     - Description

   * - ``FLD01``

     - Fluid

     - 2D channel (Poiseuille) flow in a periodic domain.

   * - ``FLD02``

     - Fluid

     - 2D Couette flow in a channel.

   * - ``FLD03``

     - Fluid

     - Pressure-driven channel flow with specified pressure boundaries.

   * - ``DEM01``

     - DEM

     - Freely falling particle with wall collision; verifies spring-dashpot model.

   * - ``DEM02``

     - DEM

     - Repeated particle bouncing; compares rebound height to theory.

   * - ``DEM03``

     - DEM

     - Two compressed particles between walls; tests multi-particle contact.

   * - ``DEM04``

     - DEM

     - Rolling/slipping particle on a rough surface; tests friction model.

   * - ``DEM05``

     - DEM

     - Oblique particle collisions with a flat wall; tests tangential restitution.

   * - ``DEM06``

     - Coupled

     - Single particle terminal velocity in a fluid; tests drag models.

   * - ``DEM07``

     - Coupled

     - Homogeneous Cooling System (HCS); tests granular temperature decay.

Details for each case are available in ``tests/README.md``.

Relationship to Regression Tests

--------------------------------

The CI pipeline **does not** run the CCSE regression suite.

* The **Regression Tests** page documents the CCSE/AMReX regression framework.

* The CI page documents the **CTest-based MFIX-Exa Test Suite** and infrastructure tests.

These two systems are complementary:

* CI ensures build correctness and basic functional behavior.

* Regression tests ensure numerical consistency across platforms and versions.

Developer Expectations

----------------------

* All CI jobs must pass before merging into ``develop``.

* Lint and warnings jobs help maintain code quality.

* Build jobs ensure portability across compilers and GPU backends.

* Test jobs ensure functional correctness.

* CI artifacts contain detailed logs for debugging failures.

.. _Chap:NightlyTesting :

Nightly Tests

=============

The following regression tests are run nightly with MFIX-Exa.   The plotfiles generated in each night's test 

are compared with the benchmark plotfiles using the AMReX :cpp:`fcompare` utility to compare the mesh data

and :cpp:`particle_compare` to compare the particle data.

The results of these tests can be found at https://ccse.lbl.gov/pub/RegressionTesting/MFIX-Exa/

Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks.

"Auto" means the particles were generated automatically with the random number

generator; if "Auto" is not specified the particle data were read in from "particle_input.dat"

These first tests have both fluid and particles and are run in rectangular geometries;

all tests except DEM06 use drag type "BVK2". 

"NSW" means "No Slip Wall" and "Per" is "periodic."

"MI/PO" refers to Mass Inflow at the low end of the domain and Pressure Outflow at the high end.

+-------------------+----+--------+------+-------+----+----+----------------------+

| Test              | nx | bc_x   | EB   | Npa   | Ng | Np | What does this test? |

|                   | ny | bc_y   |      |       |    |    |                      |

|                   | nz | bc_z   |      |       |    |    |                      |

+===================+====+========+======+=======+====+====+======================+

| BENCH01           | 32 | Per    | None | 5005  | 1  | 1  | Triply periodic      |

| Size0001          | 32 | Per    |      |       |    |    |                      |

|                   | 32 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH01           | 64 | Per    | None | 40040 | 8  | 4  | Replicate            |

| Size0001          | 64 | Per    |      |       |    |    |                      |

| replicate         | 64 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH01           | 32 | Per    | None | 5005  | 8  | 4  | Restart              |

| Size0001          | 32 | Per    |      |       |    |    |                      |

| restart           | 32 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH02           | 10 | Per    | None | 1611  | 1  | 1  | Mixed NSW / Per      |

| Size0001          | 10 | NSW    |      |       |    |    |                      |

|                   | 10 | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH02           | 10 | NSW    | None | 1611  | 1  | 1  | NSW on all faces     |

| Size0001          | 10 | NSW    |      |       |    |    |                      |

| walls             | 10 | NSW    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH03           | 4  | Per    | None | 2500  | 1  | 1  | Mixed MI/PO + Per    |

| Size0001          | 50 | MI/PO  |      |       |    |    |                      |

|                   | 4  | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| BENCH04           | 4  | Per    | None | 224   | 1  | 1  | Triply periodic      |

| Size0001          | 50 | Per    |      |       |    |    |                      |

|                   | 4  | Per    |      |       |    |    |                      |

+-------------------+----+--------+------+-------+----+----+----------------------+

| DEM06             | 5  | Per    | None | 1     | 10 | 4  | Single particle      |

| z multiple        | 5  | Per    |      |       |    |    | falling in fluid     |

|                   | 50 | MI/PO  |      |       |    |    | (user_drag)          |

+-------------------+----+--------+------+-------+----+----+----------------------+

This second set of tests have both fluid and particles and are run in cylindrial geometries

interior to the domain boundaries; they also use drag type "BVK2".  Here "IGN" means

those domain boundaries should be ignored because they are outside the EB boundary.

+-------------------+----+-------+------+--------+----+----+----------------------+

| Test              | nx | bc_x  | EB   | Npa    | Ng | Np | What does this test? |

|                   | ny | bc_y  |      |        |    |    |                      |

|                   | nz | bc_z  |      |        |    |    |                      |

+===================+====+=======+======+========+====+====+======================+

| BENCH05           | 40 | MI/PO | Cyl  | 7949   | 4  | 4  | EB in parallel       |

| Size0008          | 10 | IGN   |      | Auto   |    |    |                      |

|                   | 10 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 7968   | 4  | 1  | EB in serial         |

| Size0008          | 10 | IGN   |      | Auto   |    |    |                      |

| serial            | 10 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 36672  | 16 | 4  | Regrid & dual grid   |

| Size0008          | 20 | IGN   |      | Auto   |    |    |                      |

| medium            | 20 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH05           | 40 | MI/PO | Cyl  | 157106 | 16 | 4  | Regrid & dual grid   |

| Size0008          | 40 | IGN   |      | Auto   |    |    |                      |

| wide              | 40 | IGN   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH06           | 40 | Per   | Cyl  | 627    | 4  | 1  | EB                   |

| Size0008          | 10 | IGN   |      | Auto   |    |    | with periodic        |

| serial            | 10 | IGN   |      |        |    |    | serial               |

+-------------------+----+-------+------+--------+----+----+----------------------+

| BENCH06           | 40 | Per   | Cyl  | 624    | 4  | 4  | EB                   |

| Size0008          | 10 | IGN   |      | Auto   |    |    | with periodic        |

|                   | 10 | IGN   |      |        |    |    | parallel             |

+-------------------+----+-------+------+--------+----+----+----------------------+

This third set of tests is particles-only in rectangular geometries.

+-------------------+----+-------+------+--------+----+----+----------------------+

| Test              | nx | bc_x  | EB   | Npa    | Ng | Np | What does this test? |

|                   | ny | bc_y  |      |        |    |    |                      |

|                   | nz | bc_z  |      |        |    |    |                      |

+===================+====+=======+======+========+====+====+======================+

| DEM01             | 4  | NSW   | None |   1    | 1  | 1  | Particle  only       |

| x single          | 4  | NSW   |      |        |    |    |                      |

|                   | 4  | NSW   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| DEM03             | 5  | Per   | None |   2    | 1  | 1  | Particles only       |

| z single          | 5  | Per   |      |        |    |    |                      |

|                   | 2  | NSW   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

| DEM04             | 4  | NSW   | None |   1    | 1  | 1  | Particles only       |

| z single          | 4  | Per   |      |        |    |    |                      |

|                   | 4  | Per   |      |        |    |    |                      |

+-------------------+----+-------+------+--------+----+----+----------------------+

Running the MFIX-Exa Test Suite

===============================

MFIX-Exa comes with several tests aimed at evaluating software

functionalities. The source files as well as the required input files

for each test are located in the ``tests`` directory. The ``tests``

directory is copied to the build directory during MFIX-Exa configuration

process. When a test is run (see below), output files are stored in

``build_dir/tests/test-name``.

There are various dependencies for comparing test results.

o To compare results to archived flow slices stored in ``AUTOTEST``

directories with the case files, the environment variable ``FEXTRACT``

must point to the location of the AMReX ``fextract`` utility located in

the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally,

``numdiff`` must be installed for comparing results.

o To compare point-by-point field data, the environment variable

``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids``

found in the directory, ``amrex/Tools/PostProcessing/F_Src``.

Additionally, the environment variable ``MFIX_BENCHMARKS_HOME`` must

point to the location of a local regression test data set. See

*Generating local regression test data* for instructions on creating a

local regression test data set.

Run all tests

-------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest

List all tests (without running them)

-------------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -N

Run a particular test by the index listed in ctest -N

-----------------------------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -I 3,3             # run the third test

Run a particular test by name

-----------------------------

.. code:: shell

    > cd to mfix-build-dir

    > ctest -R DEM01  # running all tests with "DEM01" in the test name

Run a particular test via make

------------------------------

.. code:: shell

    > cd to mfix-build-dir

    > make run_DEM01-x  # running "DEM01-x" and output to the screen

Run specific

------------

If the environment variable GRID is defined, it specifies which grid

types to run for the test(s). If GRID variable is not defined, the

default is to run the tests for all grid types. > env GRID="tiled" ctest

-R DEM01 # running all tests with "DEM01" for tiled grid > env

GRID="single multiple" ctest -R DEM01 # running all tests with "DEM01"

for single grid and multiple grid > ctest -R DEM01 # running all tests

with "DEM01" for all grid types (single, multiple, tiled)

Run a user-defined case

-----------------------

.. code:: shell

    > ./mfix inputs-myrun

*inputs-myrun* is a text file containing the AMReX input parameters;

this can be named anything as long as it is the **first** argument

following the executable. Note that many of the problem parameters are

still defined in *mfix.dat*.