.. _Chap:RegressionTesting: Regression Testing ================== The MFIX-Exa regression test suite is built on top of the AMReX regression testing framework, which is developed and maintained by the AMReX team. The framework provides tools for building applications, running benchmark problems, and comparing results against reference datasets. The AMReX regression testing system is open source and available at: https://github.com/AMReX-Codes/regression_testing This page describes how to configure and run the regression tests, how to customize the test configuration files, and provides a summary of the available benchmark problems. Overview -------- Regression tests are used to ensure that changes to MFIX-Exa do not introduce unintended differences in numerical results. These tests: * build MFIX-Exa and AMReX in a controlled configuration, * run a suite of benchmark problems, * compare the results against reference data, * report any differences. The regression suite is **not** part of the CI pipeline. CI runs the CTest-based MFIX-Exa Test Suite, while regression tests must be run manually by developers. Regression Test Configuration Files ----------------------------------- MFIX-Exa provides **two separate regression test configuration files**, located in the ``tests/`` directory: .. code-block:: text tests/MFIX-tests-CPU.ini tests/MFIX-tests-GPU.ini Use the CPU configuration file for CPU-only regression testing and the GPU configuration file for CUDA-enabled regression testing. At this time, MFIX-Exa does not test configurations for HIP (AMD) or SYCL (Intel) backends. For GPU (CUDA) regression tests, the benchmark datasets are generated using a CPU-only build by setting ``-DAMReX_GPU_BACKEND=NONE`` and ``-DMFIX_GPU_BACKEND=NONE``. This ensures that CPU and GPU runs are compared against a common reference dataset. Because GPU runs may exhibit small numerical differences relative to CPU-generated benchmarks, the regression tolerances for GPU tests are typically less strict. Structure of the Configuration Files ------------------------------------ Each ``.ini`` file contains several sections: * ``[main]`` – global settings for the regression framework * ``[AMReX]`` – AMReX source location and CMake options * ``[source]`` – MFIX-Exa source location and CMake options * ``[BENCHXX-*]`` – individual benchmark problem definitions Only a small number of entries typically need to be modified by users. User-Editable Fields -------------------- Required Edits ~~~~~~~~~~~~~~ Users must update the following fields in the ``ini`` file. ``MFIX-tests-GPU.ini``: * ``testTopDir`` – directory where regression outputs will be written. * ``webTopDir`` – directory for HTML reports (optional). * ``[AMReX]/dir`` – path to the AMReX source tree, typically the version located in the ``subprojects/amrex`` directory. * ``[source]/dir`` – path to the MFIX-Exa source tree. Example: .. code-block:: ini testTopDir = /path/to/regression/output webTopDir = /path/to/regression/web [AMReX] dir = /path/to/mfix-exa/subprojects/amrex [source] dir = /path/to/mfix-exa When maintaining both CPU and GPU regression suites on the same system, the ``testTopDir`` and ``webTopDir`` entries in each ``ini`` file should point to different directories. This ensures that the benchmark datasets and test results for CPU and GPU runs remain separate. Mixing these directories can lead to incorrect comparisons, overwritten benchmark files, or misleading regression reports. Optional Edits ~~~~~~~~~~~~~~ These entries may be modified depending on the system: * ``MPIcommand`` – command used to launch MPI jobs * ``MPIhost`` – hostname for MPI execution (optional) * ``numMakeJobs`` – number of parallel build jobs * ``branch`` – AMReX or MFIX-Exa branch to check out Advanced Edits ~~~~~~~~~~~~~~ These entries are rarely modified: * ``cmakeSetupOpts`` in ``[AMReX]`` or ``[source]`` * ``runtime_params`` inside individual benchmark sections * ``numprocs`` or ``numthreads`` for scaling studies Running the Regression Tests ---------------------------- The MFIX-Exa regression suite is driven by the AMReX regression testing framework. Before creating or running any regression tests, ensure that the directories specified in the ``ini`` configuration file exist: * ``testTopDir`` – directory where benchmark and test output will be written * ``webTopDir`` – directory where HTML reports will be generated Both directories must be created manually before running the regression framework. In the examples below, ``regression_testing`` refers to the AMReX regression testing repository available at: https://github.com/AMReX-Codes/regression_testing Creating Benchmark Datasets ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Benchmark datasets serve as the reference results against which future runs are compared. They must be created before running the regression suite for the first time, or whenever intentional changes to MFIX-Exa require updated reference results. Create benchmark datasets for **all** tests listed in the ``ini`` file: .. code-block:: bash ./regression_testing/regtest.py \ --make_benchmarks "creating initial benchmark datasets" \ MFIX-tests.ini Create benchmark datasets for a **subset** of tests: .. code-block:: bash ./regression_testing/regtest.py \ --make_benchmarks "creating initial benchmark datasets" \ --tests 'LIST OF BENCHMARK CASES' \ MFIX-tests.ini Running the Regression Tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To run the regression suite without updating AMReX or MFIX-Exa, use the ``--no_update all`` flag. This prevents the framework from checking out the branches listed in the ``ini`` file and is recommended when testing local modifications. Run specific tests with a descriptive note: .. code-block:: bash ./regression_testing/regtest.py \ --no_update all \ --note "test after making change XYZ" \ --tests 'BENCH01-Size0001 BENCH02-Size0001' \ MFIX-tests.ini Updating Benchmark Results ~~~~~~~~~~~~~~~~~~~~~~~~~~ If a code modification is known to change numerical results, the benchmark datasets must be updated. There are two ways to do this: 1. **Recreate benchmarks using ``--make_benchmarks``** (recommended when changes are intentional and validated) 2. **Copy benchmark results from a failed test run** (useful when the regression suite has already produced new output) Update benchmarks using the ``copy_benchmarks`` option: .. code-block:: bash ./regression_testing/regtest.py \ --no_update all \ --copy_benchmarks "Updating after submodules" \ MFIX-tests.ini This replaces the existing benchmark plotfiles with those generated during the most recent test run. Interpreting Results -------------------- After the tests complete, the framework generates: * a summary of passed/failed tests, * detailed logs for each benchmark, * optional HTML reports (if ``webTopDir`` is set), * particle comparison results (if ``compareParticles = 1``). A test is considered failed if: * MFIX-Exa output differs from the benchmark data, * the simulation crashes or produces invalid output, * particle trajectories differ beyond tolerance. Benchmark Summary ----------------- The MFIX-Exa regression suite includes a broad set of benchmark problems covering fluid flow, particle dynamics, and coupled multiphase physics. A high-level summary is provided below. .. list-table:: :header-rows: 1 :widths: 15 20 65 * - Benchmark - Category - Description * - ``BENCH01`` - HCS - Homogeneous Cooling System (HCS) with multiple advection and redistribution options; includes restart and replicate tests. * - ``BENCH02`` - Settling - Single-particle settling in fluid; includes no-fluid and wall-bounded variants. * - ``BENCH03`` - Fluidized bed - Small fluidized bed with gas–solid interaction. * - ``BENCH04`` - Square riser - Gas–solid riser flow in a square duct. * - ``BENCH05`` - Cylindrical fluidized bed - Multiple grid sizes and physics options; includes chemistry and covered-grid tests. * - ``BENCH06`` - Cylindrical riser - Gas–solid riser with multiple MPI configurations. * - ``BENCH08`` - PIC - Particle-in-cell tests including restart variants. * - ``BENCH09`` - Refinement - AMR refinement test for particle and fluid fields. * - ``DEM01–DEM06`` - DEM - Particle-only verification tests (free fall, bouncing, compression, rolling, oblique collisions, terminal velocity). * - ``TracerAdv`` - Tracer transport - Tracer advection with optional Godunov/MOL advection schemes. Relationship to CI Tests ------------------------ The regression suite is **separate** from the MFIX-Exa CI pipeline: * CI runs the **CTest-based MFIX-Exa Test Suite**. * Regression tests use the **AMReX regression framework**. * CI tests focus on build correctness and basic functionality. * Regression tests focus on numerical correctness and reproducibility. Both systems complement each other and should be used regularly during development. Developer Expectations ---------------------- * CPU and GPU regression tests should be run when modifying multiphase physics, AMReX interfaces, or GPU kernels. * Any regression failure must be investigated and resolved before merging. * Benchmark data should only be updated when numerical changes are intentional and documented.