diff --git a/docs/source_docs/index.rst b/docs/source_docs/index.rst index fd5c22545597b6741cd81e29740549fcff64dea4..2425d5a9b0fbe647fb864990e59a09d83c47eff1 100644 --- a/docs/source_docs/index.rst +++ b/docs/source_docs/index.rst @@ -78,10 +78,8 @@ and performance: :maxdepth: 0 :caption: Tests and Benchmarks: - test_benchmarks/RunningTestSuite - test_benchmarks/RegressionTesting test_benchmarks/CITests - test_benchmarks/NightlyTests + test_benchmarks/RegressionTesting test_benchmarks/qualitative_bencharks/index test_benchmarks/verification/index diff --git a/docs/source_docs/test_benchmarks/CITests.rst b/docs/source_docs/test_benchmarks/CITests.rst index 35a3cf22ee182a330caf06547c016c2e0f5d211c..40ccb5842c28e7917c6c910deaa487ac0d8cb035 100644 --- a/docs/source_docs/test_benchmarks/CITests.rst +++ b/docs/source_docs/test_benchmarks/CITests.rst @@ -1,107 +1,243 @@ -.. _Chap:CITesting : - -Continuous Integration -====================== - -The following regression tests are run every time a commit is pushed to the main -MFIX-Exa repository on the NETL gitlab. - -For each of the tests in the chart below, there are -three directional variations; these are identified in the repository as, -for example, FLD01-x, FLD01-y, and FLD01-z. - -For each direction, where appropriate, there are multiple versions, with the following notations: - - * SGS: single grid serial - - * MGS: multiple grid serial - - * TGS: tiled grid serial - - * MGP: multiple grid parallel - -Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks. - -All the FLD cases are fluid-only and steady state. -All the DEM cases are particle-only except for DEM06 and DEM07 which are fluid and particles; -these both use the "BVK2" drag type. -In all cases the particle data were read in from "particle_input.dat" - -None of these tests have non-rectangular geometry. - -"NSW" means "No Slip Wall" and "Per" is "periodic." -"MI/PO" refers to Mass Inflow at the low end of the domain and Pressure Outflow at the high end. -"PI/PO" refers to Pressure Inflow at the low end of the domain and Pressure Outflow at the high end. - -Additional detail about these problems is given in tests/README.md - -Single-grid, single-process (SGS) particle-only tests: - -+-------+----+----+----+------+------+-------+-----+--------------------+ -| Test | nx | ny | nz | bc_x | bc_y | bc_z | Npa | Description | -+=======+====+====+====+======+======+=======+=====+====================+ -| DEM01 | 2 | 5 | 5 | NSW | Per | Per | 1 | Freely falling | -| | | | | | | | | particle with | -| | | | | | | | | wall collision | -+-------+----+----+----+------+------+-------+-----+--------------------+ -| DEM02 | 2 | 5 | 5 | NSW | Per | Per | 1 | Multiple bounces | -| | | | | | | | | with bounce height | -| | | | | | | | | measured | -+-------+----+----+----+------+------+-------+-----+--------------------+ -| DEM03 | 2 | 5 | 5 | NSW | Per | Per | 2 | Two stacked | -| | | | | | | | | compressed | -| | | | | | | | | particles | -+-------+----+----+----+------+------+-------+-----+--------------------+ -| DEM04 | 4 | 4 | 4 | NSW | Per | Per | 1 | Single particle | -| | | | | | | | | slipping on a | -| | | | | | | | | rough surface | -+-------+----+----+----+------+------+-------+-----+--------------------+ -| DEM05 | 5 | 2 | 5 | Per | Per | Per | 93 | Oblique particle | -| | | | | | | | | collisions | -| | | | | | | | | | -+-------+----+----+----+------+------+-------+-----+--------------------+ - - -Steady-state fluid-only tests: - -+-------+-----+----+----+----+-------+------+------+-----+----------------------+ -| Test | | nx | ny | nz | bc_x | bc_y | bc_z | Ng | Np | -+=======+=====+====+====+====+=======+======+======+=====+======================+ -| FLD01 | | 8 | 8 | 4 | Per | NSW | Per | Poiseuille flow | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ -| | SGS | | | | | | | 1 | 1 | | -| | MGS | | | | | | | 4 | 1 | | -| | MGP | | | | | | | 4 | 8 | | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ -| FLD02 | | 80 |16 | 16 | MI/PO | NSW | NSW | Couette flow | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ -| | SGS | | | | | | | 1 | 1 | | -| | MGS | | | | | | | 40 | 1 | | -| | MGP | | | | | | | 40 | 8 | | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ -| FLD03 | | 8 | 8 | 4 | PI/PO | NSW | Per | Poiseuille flow | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ -| | SGS | | | | | | | 1 | 1 | | -| | MGS | | | | | | | 4 | 1 | | -| | MGP | | | | | | | 4 | 8 | | -+-------+-----+----+----+----+-------+------+------+-----+----+-----------------+ - -Coupled particle/fluid tests: - -+-------+-----+----+----+----+------+------+------+------+----+--------------------+ -| Test | | nx | ny | nz | bc_x | bc_y | bc_z | Npa | Ng | Np | -+=======+=====+====+====+====+======+======+======+======+====+====================+ -| DEM06 | | 50 | 5 | 5 | NSW | NSW | NSW | 1 | Single particle falling | -| | | | | | | | | | under gravity | -+-------+-----+----+----+----+------+------+------+------+----+----+---------------+ -| | SGS | | | | | | | | 1 | 1 | | -| | MGS | | | | | | | | 10 | 1 | | -| | MGP | | | | | | | | 10 | 8 | | -+-------+-----+----+----+----+------+------+------+------+----+----+---------------+ -| DEM07 | | 20 | 20 | 20 | Per | Per | Per | 1222 | Homogeneous cooling | -| | | | | | | | | | system | -+-------+-----+----+----+----+------+------+------+------+----+----+---------------+ -| | SGS | | | | | | | | 1 | 1 | | -| | MGS | | | | | | | | 8 | 1 | | -| | MGP | | | | | | | | 8 | 8 | | -+-------+-----+----+----+----+------+------+------+------+----+----+---------------+ +.. _Chap:CITesting: + +Continuous Integration (CI) +=========================== + +MFIX-Exa uses a GitLab-based Continuous Integration (CI) pipeline to ensure +code quality, portability, and correctness across multiple compilers, GPU +backends, and build configurations. This page describes what the CI runs, +how to reproduce CI behavior locally, and provides a high-level overview of +the MFIX-Exa test suite. + + +When the CI Pipeline Runs +------------------------- + +The CI pipeline is triggered under the following conditions: + +* **Merge requests** – runs linting, warnings builds, and multi-backend builds. +* **Commits to the ``develop`` branch** – runs the full test pipeline. +* **Tag pushes** – used for releases. +* **Manual runs** – triggered from the GitLab web interface. + + +CI Pipeline Structure +--------------------- + +The MFIX-Exa CI is organized into two distinct execution paths: + +* **Merge Request (MR) pipelines** – run on every push to a merge request. + These pipelines focus on code quality, portability, and build correctness. + They include the **Lint**, **Warnings**, and **Build** stages. + +* **Post-merge pipelines** – run only after changes are merged into the + ``develop`` branch. These pipelines run the full **CTest-based test suite** + and the specialized **Ascent** and **MPMD** tests. + +This separation ensures fast feedback during code review while reserving +the more expensive tests for post-merge validation. + + +Merge Request Pipeline Stages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Lint** + + Static analysis and formatting checks: + + * Shell scripts (``shellcheck``) + * Spelling (``codespell``) + * Tabs and trailing whitespace removal + * Python linting (``pylint``) for the GUI + +**Warnings Builds** + + Compiler-strict builds using GCC and Clang in both Debug and Release modes. + These builds enable extensive warning flags (``-Wall``, ``-Wextra``, + ``-Werror``, etc.) to catch portability and correctness issues early. + +**Builds** + + Multiple configurations are compiled to ensure portability across compilers + and GPU backends: + + * CPU-only builds + * GPU-enabled builds (CUDA, HIP, SYCL) + * CSG-enabled builds + * MPMD-enabled builds + * Unit tests + +These MR-stage jobs verify that MFIX-Exa builds cleanly and consistently +before code is merged. + + +Post-Merge Test Pipeline +~~~~~~~~~~~~~~~~~~~~~~~~ + +After changes are merged into the ``develop`` branch, the CI runs the full +MFIX-Exa test suite. These jobs are triggered by ``rules:`` blocks that +restrict execution to the ``develop`` branch. + +The post-merge pipeline includes: + +**CTest-based MFIX-Exa Test Suite** + + Tests are grouped by label: + + * ``SGS`` – single-grid serial tests + * ``MGS`` – multi-grid serial tests + * ``TGS`` – tiled-grid serial tests + * ``MGP`` – multi-grid parallel tests + * ``TGPO`` – tiled-grid parallel OpenMP tests + * ``TGSO`` – tiled-grid serial OpenMP tests + +**Ascent Visualization Test** + + Runs a small HCS benchmark with Ascent enabled and verifies image output. + +**MPMD Tutorial Test** + + Runs the MPMD tutorial example and checks numerical output and generated + images. + + +How to Reproduce CI Locally +--------------------------- + +Building MFIX-Exa +~~~~~~~~~~~~~~~~~ + +A typical local build mirrors the CI configuration. For example: + +.. code-block:: bash + + cmake -S -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -G Ninja + + cmake --build build --target build_tests unit_tests + +Additional options (e.g., enabling MPI, OpenMP, GPU backends, or CSG) +can be added as needed. + + +Running the MFIX-Exa Test Suite +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +CTest is used to run the MFIX-Exa test suite. All tests are available +locally, regardless of which subset the CI runs. + +List all tests: + +.. code-block:: bash + + ctest --test-dir build/ -N + +Run all tests: + +.. code-block:: bash + + ctest --test-dir build/ + +Run a specific test by name: + +.. code-block:: bash + + ctest --test-dir build/ -R DEM01 + +Run a specific test by index: + +.. code-block:: bash + + ctest --test-dir build/ -I 3,3 + +Run selected the signle-grid serial DEM tests in the ``y`` direction: + +.. code-block:: bash + + ctest --test-dir build/ -R "DEM(01|03|04|05|06)-y_SGS|DEM07_SGS" + + +Running Unit Tests +~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + cmake --build build --target unit_tests + ctest --test-dir build/ -R unit_test -V + + +Summary of MFIX-Exa Test Categories +----------------------------------- + +The MFIX-Exa test suite includes fluid-only tests, particle-only tests, +and coupled fluid/particle tests. The CI runs a representative subset of +these, while the full suite is available locally. + +Below is a high-level summary of the test categories and representative +cases. + +.. list-table:: + :header-rows: 1 + :widths: 15 20 65 + + * - Test + - Type + - Description + * - ``FLD01`` + - Fluid + - 2D channel (Poiseuille) flow in a periodic domain. + * - ``FLD02`` + - Fluid + - 2D Couette flow in a channel. + * - ``FLD03`` + - Fluid + - Pressure-driven channel flow with specified pressure boundaries. + * - ``DEM01`` + - DEM + - Freely falling particle with wall collision; verifies spring-dashpot model. + * - ``DEM02`` + - DEM + - Repeated particle bouncing; compares rebound height to theory. + * - ``DEM03`` + - DEM + - Two compressed particles between walls; tests multi-particle contact. + * - ``DEM04`` + - DEM + - Rolling/slipping particle on a rough surface; tests friction model. + * - ``DEM05`` + - DEM + - Oblique particle collisions with a flat wall; tests tangential restitution. + * - ``DEM06`` + - Coupled + - Single particle terminal velocity in a fluid; tests drag models. + * - ``DEM07`` + - Coupled + - Homogeneous Cooling System (HCS); tests granular temperature decay. + + +Details for each case are available in ``tests/README.md``. + + +Relationship to Regression Tests +-------------------------------- + +The CI pipeline **does not** run the CCSE regression suite. + +* The **Regression Tests** page documents the CCSE/AMReX regression framework. +* The CI page documents the **CTest-based MFIX-Exa Test Suite** and infrastructure tests. + +These two systems are complementary: + +* CI ensures build correctness and basic functional behavior. +* Regression tests ensure numerical consistency across platforms and versions. + + +Developer Expectations +---------------------- + +* All CI jobs must pass before merging into ``develop``. +* Lint and warnings jobs help maintain code quality. +* Build jobs ensure portability across compilers and GPU backends. +* Test jobs ensure functional correctness. +* CI artifacts contain detailed logs for debugging failures. diff --git a/docs/source_docs/test_benchmarks/NightlyTests.rst b/docs/source_docs/test_benchmarks/NightlyTests.rst deleted file mode 100644 index 81fea01888f404e8673853b2a302034ef34ec624..0000000000000000000000000000000000000000 --- a/docs/source_docs/test_benchmarks/NightlyTests.rst +++ /dev/null @@ -1,114 +0,0 @@ -.. _Chap:NightlyTesting : - -Nightly Tests -============= - -The following regression tests are run nightly with MFIX-Exa. The plotfiles generated in each night's test -are compared with the benchmark plotfiles using the AMReX :cpp:`fcompare` utility to compare the mesh data -and :cpp:`particle_compare` to compare the particle data. - -The results of these tests can be found at https://ccse.lbl.gov/pub/RegressionTesting/MFIX-Exa/ - -Below Ng = number of grids, Npa = number of particles, Np = number of MPI ranks. - -"Auto" means the particles were generated automatically with the random number -generator; if "Auto" is not specified the particle data were read in from "particle_input.dat" - -These first tests have both fluid and particles and are run in rectangular geometries; -all tests except DEM06 use drag type "BVK2". - -"NSW" means "No Slip Wall" and "Per" is "periodic." -"MI/PO" refers to Mass Inflow at the low end of the domain and Pressure Outflow at the high end. - -+-------------------+----+--------+------+-------+----+----+----------------------+ -| Test | nx | bc_x | EB | Npa | Ng | Np | What does this test? | -| | ny | bc_y | | | | | | -| | nz | bc_z | | | | | | -+===================+====+========+======+=======+====+====+======================+ -| BENCH01 | 32 | Per | None | 5005 | 1 | 1 | Triply periodic | -| Size0001 | 32 | Per | | | | | | -| | 32 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH01 | 64 | Per | None | 40040 | 8 | 4 | Replicate | -| Size0001 | 64 | Per | | | | | | -| replicate | 64 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH01 | 32 | Per | None | 5005 | 8 | 4 | Restart | -| Size0001 | 32 | Per | | | | | | -| restart | 32 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH02 | 10 | Per | None | 1611 | 1 | 1 | Mixed NSW / Per | -| Size0001 | 10 | NSW | | | | | | -| | 10 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH02 | 10 | NSW | None | 1611 | 1 | 1 | NSW on all faces | -| Size0001 | 10 | NSW | | | | | | -| walls | 10 | NSW | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH03 | 4 | Per | None | 2500 | 1 | 1 | Mixed MI/PO + Per | -| Size0001 | 50 | MI/PO | | | | | | -| | 4 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| BENCH04 | 4 | Per | None | 224 | 1 | 1 | Triply periodic | -| Size0001 | 50 | Per | | | | | | -| | 4 | Per | | | | | | -+-------------------+----+--------+------+-------+----+----+----------------------+ -| DEM06 | 5 | Per | None | 1 | 10 | 4 | Single particle | -| z multiple | 5 | Per | | | | | falling in fluid | -| | 50 | MI/PO | | | | | (user_drag) | -+-------------------+----+--------+------+-------+----+----+----------------------+ - -This second set of tests have both fluid and particles and are run in cylindrial geometries -interior to the domain boundaries; they also use drag type "BVK2". Here "IGN" means -those domain boundaries should be ignored because they are outside the EB boundary. - -+-------------------+----+-------+------+--------+----+----+----------------------+ -| Test | nx | bc_x | EB | Npa | Ng | Np | What does this test? | -| | ny | bc_y | | | | | | -| | nz | bc_z | | | | | | -+===================+====+=======+======+========+====+====+======================+ -| BENCH05 | 40 | MI/PO | Cyl | 7949 | 4 | 4 | EB in parallel | -| Size0008 | 10 | IGN | | Auto | | | | -| | 10 | IGN | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| BENCH05 | 40 | MI/PO | Cyl | 7968 | 4 | 1 | EB in serial | -| Size0008 | 10 | IGN | | Auto | | | | -| serial | 10 | IGN | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| BENCH05 | 40 | MI/PO | Cyl | 36672 | 16 | 4 | Regrid & dual grid | -| Size0008 | 20 | IGN | | Auto | | | | -| medium | 20 | IGN | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| BENCH05 | 40 | MI/PO | Cyl | 157106 | 16 | 4 | Regrid & dual grid | -| Size0008 | 40 | IGN | | Auto | | | | -| wide | 40 | IGN | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| BENCH06 | 40 | Per | Cyl | 627 | 4 | 1 | EB | -| Size0008 | 10 | IGN | | Auto | | | with periodic | -| serial | 10 | IGN | | | | | serial | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| BENCH06 | 40 | Per | Cyl | 624 | 4 | 4 | EB | -| Size0008 | 10 | IGN | | Auto | | | with periodic | -| | 10 | IGN | | | | | parallel | -+-------------------+----+-------+------+--------+----+----+----------------------+ - - -This third set of tests is particles-only in rectangular geometries. - -+-------------------+----+-------+------+--------+----+----+----------------------+ -| Test | nx | bc_x | EB | Npa | Ng | Np | What does this test? | -| | ny | bc_y | | | | | | -| | nz | bc_z | | | | | | -+===================+====+=======+======+========+====+====+======================+ -| DEM01 | 4 | NSW | None | 1 | 1 | 1 | Particle only | -| x single | 4 | NSW | | | | | | -| | 4 | NSW | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| DEM03 | 5 | Per | None | 2 | 1 | 1 | Particles only | -| z single | 5 | Per | | | | | | -| | 2 | NSW | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ -| DEM04 | 4 | NSW | None | 1 | 1 | 1 | Particles only | -| z single | 4 | Per | | | | | | -| | 4 | Per | | | | | | -+-------------------+----+-------+------+--------+----+----+----------------------+ diff --git a/docs/source_docs/test_benchmarks/RegressionTesting.rst b/docs/source_docs/test_benchmarks/RegressionTesting.rst index 4740fa1a134433ec01b70cb0261a71f978dc2892..42ee2b68c0466195d3c7dbfa3902cffb01262f6c 100644 --- a/docs/source_docs/test_benchmarks/RegressionTesting.rst +++ b/docs/source_docs/test_benchmarks/RegressionTesting.rst @@ -1,75 +1,304 @@ -Regression testing +.. _Chap:RegressionTesting: + +Regression Testing ================== -Generating local regression test data -------------------------------------- +The MFIX-Exa regression test suite is built on top of the AMReX +regression testing framework, which is developed and maintained by the +AMReX team. The framework provides tools for building applications, +running benchmark problems, and comparing results against reference +datasets. The AMReX regression testing system is open source and +available at: + + https://github.com/AMReX-Codes/regression_testing + + +This page describes how to configure and run the regression tests, how +to customize the test configuration files, and provides a summary of the +available benchmark problems. + + +Overview +-------- + +Regression tests are used to ensure that changes to MFIX-Exa do not +introduce unintended differences in numerical results. These tests: + +* build MFIX-Exa and AMReX in a controlled configuration, +* run a suite of benchmark problems, +* compare the results against reference data, +* report any differences. + +The regression suite is **not** part of the CI pipeline. CI runs the +CTest-based MFIX-Exa Test Suite, while regression tests must be run +manually by developers. + + +Regression Test Configuration Files +----------------------------------- + +MFIX-Exa provides **two separate regression test configuration +files**, located in the ``tests/`` directory: + +.. code-block:: text + + tests/MFIX-tests-CPU.ini + tests/MFIX-tests-GPU.ini + + +Use the CPU configuration file for CPU-only regression testing and the GPU +configuration file for CUDA-enabled regression testing. At this time, +MFIX-Exa does not test configurations for HIP (AMD) or SYCL (Intel) backends. + +For GPU (CUDA) regression tests, the benchmark datasets are generated using a +CPU-only build by setting ``-DAMReX_GPU_BACKEND=NONE`` and +``-DMFIX_GPU_BACKEND=NONE``. This ensures that CPU and GPU runs are +compared against a common reference dataset. Because GPU runs may exhibit +small numerical differences relative to CPU-generated benchmarks, the +regression tolerances for GPU tests are typically less strict. + + +Structure of the Configuration Files +------------------------------------ + +Each ``.ini`` file contains several sections: + +* ``[main]`` – global settings for the regression framework +* ``[AMReX]`` – AMReX source location and CMake options +* ``[source]`` – MFIX-Exa source location and CMake options +* ``[BENCHXX-*]`` – individual benchmark problem definitions + +Only a small number of entries typically need to be modified by users. + + +User-Editable Fields +-------------------- + +Required Edits +~~~~~~~~~~~~~~ + +Users must update the following fields in the ``ini`` file. +``MFIX-tests-GPU.ini``: + +* ``testTopDir`` – directory where regression outputs will be written. +* ``webTopDir`` – directory for HTML reports (optional). +* ``[AMReX]/dir`` – path to the AMReX source tree, typically the version + located in the ``subprojects/amrex`` directory. +* ``[source]/dir`` – path to the MFIX-Exa source tree. + + +Example: + +.. code-block:: ini + + testTopDir = /path/to/regression/output + webTopDir = /path/to/regression/web + [AMReX] + dir = /path/to/mfix-exa/subprojects/amrex + [source] + dir = /path/to/mfix-exa + + +When maintaining both CPU and GPU regression suites on the same system, +the ``testTopDir`` and ``webTopDir`` entries in each ``ini`` file should +point to different directories. This ensures that the benchmark datasets +and test results for CPU and GPU runs remain separate. Mixing these +directories can lead to incorrect comparisons, overwritten benchmark +files, or misleading regression reports. + + +Optional Edits +~~~~~~~~~~~~~~ + +These entries may be modified depending on the system: + +* ``MPIcommand`` – command used to launch MPI jobs +* ``MPIhost`` – hostname for MPI execution (optional) +* ``numMakeJobs`` – number of parallel build jobs +* ``branch`` – AMReX or MFIX-Exa branch to check out + + +Advanced Edits +~~~~~~~~~~~~~~ + +These entries are rarely modified: + +* ``cmakeSetupOpts`` in ``[AMReX]`` or ``[source]`` +* ``runtime_params`` inside individual benchmark sections +* ``numprocs`` or ``numthreads`` for scaling studies + + +Running the Regression Tests +---------------------------- + +The MFIX-Exa regression suite is driven by the AMReX regression testing +framework. Before creating or running any regression tests, ensure that +the directories specified in the ``ini`` configuration file exist: + +* ``testTopDir`` – directory where benchmark and test output will be written +* ``webTopDir`` – directory where HTML reports will be generated + +Both directories must be created manually before running the regression +framework. + +In the examples below, ``regression_testing`` refers to the AMReX +regression testing repository available at: + + https://github.com/AMReX-Codes/regression_testing + + +Creating Benchmark Datasets +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Benchmark datasets serve as the reference results against which future +runs are compared. They must be created before running the regression +suite for the first time, or whenever intentional changes to MFIX-Exa +require updated reference results. + +Create benchmark datasets for **all** tests listed in the ``ini`` file: + +.. code-block:: bash + + ./regression_testing/regtest.py \ + --make_benchmarks "creating initial benchmark datasets" \ + MFIX-tests.ini + +Create benchmark datasets for a **subset** of tests: + +.. code-block:: bash + + ./regression_testing/regtest.py \ + --make_benchmarks "creating initial benchmark datasets" \ + --tests 'LIST OF BENCHMARK CASES' \ + MFIX-tests.ini + + +Running the Regression Tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To run the regression suite without updating AMReX or MFIX-Exa, use the +``--no_update all`` flag. This prevents the framework from checking out +the branches listed in the ``ini`` file and is recommended when testing +local modifications. + +Run specific tests with a descriptive note: + +.. code-block:: bash + + ./regression_testing/regtest.py \ + --no_update all \ + --note "test after making change XYZ" \ + --tests 'BENCH01-Size0001 BENCH02-Size0001' \ + MFIX-tests.ini + + +Updating Benchmark Results +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If a code modification is known to change numerical results, the +benchmark datasets must be updated. There are two ways to do this: + +1. **Recreate benchmarks using ``--make_benchmarks``** + (recommended when changes are intentional and validated) + +2. **Copy benchmark results from a failed test run** + (useful when the regression suite has already produced new output) + +Update benchmarks using the ``copy_benchmarks`` option: -Developers are encouraged to create local benchmark data for regression -testing prior to committing code the GitLab repository. +.. code-block:: bash -1. Create a location to store benchmark data and clone the MFIX-Exa and - AMReX repositories. + ./regression_testing/regtest.py \ + --no_update all \ + --copy_benchmarks "Updating after submodules" \ + MFIX-tests.ini - .. code:: shell +This replaces the existing benchmark plotfiles with those generated +during the most recent test run. - mkdir /home/user/exa-rt - mkdir /home/user/exa-rt/benchmark - cd /home/user/exa-rt - git clone http://mfix.netl.doe.gov/gitlab/exa/mfix.git - git clone https://github.com/AMReX-Codes/amrex.git -2. Create a local copy the regression test setup file from the MFIX-Exa - repository. +Interpreting Results +-------------------- - .. code:: shell +After the tests complete, the framework generates: - cp mfix/RegressionTesting/MFiX-tests.ini MFiX-local.ini +* a summary of passed/failed tests, +* detailed logs for each benchmark, +* optional HTML reports (if ``webTopDir`` is set), +* particle comparison results (if ``compareParticles = 1``). -3. Edit the local setup file. Specify the top level directories for test - and web output under the ``[main]`` heading. +A test is considered failed if: - .. code:: shell +* MFIX-Exa output differs from the benchmark data, +* the simulation crashes or produces invalid output, +* particle trajectories differ beyond tolerance. - [main] - testTopDir = /home/user/exa-rt/benchmark - webTopDir = /home/user/exa-rt/web - Specify the AMReX source directory location under the ``[AMReX]`` - heading. +Benchmark Summary +----------------- - .. code:: shell +The MFIX-Exa regression suite includes a broad set of benchmark problems +covering fluid flow, particle dynamics, and coupled multiphase physics. +A high-level summary is provided below. - [AMReX] - dir = /home/user/exa-rt/amrex - branch = development +.. list-table:: + :header-rows: 1 + :widths: 15 20 65 - Specify the MFIX-Exa source directory location under the ``[source]`` - heading. + * - Benchmark + - Category + - Description + * - ``BENCH01`` + - HCS + - Homogeneous Cooling System (HCS) with multiple advection and redistribution options; includes restart and replicate tests. + * - ``BENCH02`` + - Settling + - Single-particle settling in fluid; includes no-fluid and wall-bounded variants. + * - ``BENCH03`` + - Fluidized bed + - Small fluidized bed with gas–solid interaction. + * - ``BENCH04`` + - Square riser + - Gas–solid riser flow in a square duct. + * - ``BENCH05`` + - Cylindrical fluidized bed + - Multiple grid sizes and physics options; includes chemistry and covered-grid tests. + * - ``BENCH06`` + - Cylindrical riser + - Gas–solid riser with multiple MPI configurations. + * - ``BENCH08`` + - PIC + - Particle-in-cell tests including restart variants. + * - ``BENCH09`` + - Refinement + - AMR refinement test for particle and fluid fields. + * - ``DEM01–DEM06`` + - DEM + - Particle-only verification tests (free fall, bouncing, compression, rolling, oblique collisions, terminal velocity). + * - ``TracerAdv`` + - Tracer transport + - Tracer advection with optional Godunov/MOL advection schemes. - .. code:: shell - [source] - dir = /home/user/exa-rt/mfix - branch = develop +Relationship to CI Tests +------------------------ -4. Run the AMReX regression test tool. The second argument is a user - supplied comment. +The regression suite is **separate** from the MFIX-Exa CI pipeline: - .. code:: shell +* CI runs the **CTest-based MFIX-Exa Test Suite**. +* Regression tests use the **AMReX regression framework**. +* CI tests focus on build correctness and basic functionality. +* Regression tests focus on numerical correctness and reproducibility. - cd /home/user/exa-rt - amrex/Tools/RegressionTesting/regtest.py --make_benchmarks "MFiX" MFiX-local.ini +Both systems complement each other and should be used regularly during +development. -+------------------------------------------------------------------------------------------------------------------------+ -| ## Prerequisite: Environment Dependencies on Joule (Joule specific) | -+------------------------------------------------------------------------------------------------------------------------+ -| For the Joule environment, load the gnu module and set environment variables first. If not on Joule, skip this step. | -+------------------------------------------------------------------------------------------------------------------------+ - .. code:: shell +Developer Expectations +---------------------- - module load gnu/6.1.0 - export CC=/nfs/apps/Compilers/GNU/6.1.0/bin/gcc - export CXX=/nfs/apps/Compilers/GNU/6.1.0/bin/g++ - export F77=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran - export FC=/nfs/apps/Compilers/GNU/6.1.0/bin/gfortran +* CPU and GPU regression tests should be run when modifying multiphase + physics, AMReX interfaces, or GPU kernels. +* Any regression failure must be investigated and resolved before merging. +* Benchmark data should only be updated when numerical changes are + intentional and documented. diff --git a/docs/source_docs/test_benchmarks/RunningTestSuite.rst b/docs/source_docs/test_benchmarks/RunningTestSuite.rst deleted file mode 100644 index fe9146793c8a6e8cb5d9cf89225198b46d62b5ee..0000000000000000000000000000000000000000 --- a/docs/source_docs/test_benchmarks/RunningTestSuite.rst +++ /dev/null @@ -1,88 +0,0 @@ -Running the MFIX-Exa Test Suite -=============================== - -MFIX-Exa comes with several tests aimed at evaluating software -functionalities. The source files as well as the required input files -for each test are located in the ``tests`` directory. The ``tests`` -directory is copied to the build directory during MFIX-Exa configuration -process. When a test is run (see below), output files are stored in -``build_dir/tests/test-name``. - -There are various dependencies for comparing test results. - -o To compare results to archived flow slices stored in ``AUTOTEST`` -directories with the case files, the environment variable ``FEXTRACT`` -must point to the location of the AMReX ``fextract`` utility located in -the directory, ``amrex/Tools/PostProcessing/F_Src``. Additionally, -``numdiff`` must be installed for comparing results. - -o To compare point-by-point field data, the environment variable -``FCOMPARE`` must point the AMReX utility ``plt_compare_diff_grids`` -found in the directory, ``amrex/Tools/PostProcessing/F_Src``. -Additionally, the environment variable ``MFIX_BENCHMARKS_HOME`` must -point to the location of a local regression test data set. See -*Generating local regression test data* for instructions on creating a -local regression test data set. - -Run all tests -------------- - -.. code:: shell - - > cd to mfix-build-dir - > ctest - -List all tests (without running them) -------------------------------------- - -.. code:: shell - - > cd to mfix-build-dir - > ctest -N - -Run a particular test by the index listed in ctest -N ------------------------------------------------------ - -.. code:: shell - - > cd to mfix-build-dir - > ctest -I 3,3 # run the third test - -Run a particular test by name ------------------------------ - -.. code:: shell - - > cd to mfix-build-dir - > ctest -R DEM01 # running all tests with "DEM01" in the test name - -Run a particular test via make ------------------------------- - -.. code:: shell - - > cd to mfix-build-dir - > make run_DEM01-x # running "DEM01-x" and output to the screen - -Run specific ------------- - -If the environment variable GRID is defined, it specifies which grid -types to run for the test(s). If GRID variable is not defined, the -default is to run the tests for all grid types. > env GRID="tiled" ctest --R DEM01 # running all tests with "DEM01" for tiled grid > env -GRID="single multiple" ctest -R DEM01 # running all tests with "DEM01" -for single grid and multiple grid > ctest -R DEM01 # running all tests -with "DEM01" for all grid types (single, multiple, tiled) - -Run a user-defined case ------------------------ - -.. code:: shell - - > ./mfix inputs-myrun - -*inputs-myrun* is a text file containing the AMReX input parameters; -this can be named anything as long as it is the **first** argument -following the executable. Note that many of the problem parameters are -still defined in *mfix.dat*.