diff --git a/docs/source_docs/user_guide/inputs/advanced.rst b/docs/source_docs/user_guide/inputs/advanced.rst index 7ce68e84db55df7bc20e16fc16c2cf0ef8627472..b650fdc3fcb8529978e0685044ef2aaaa1e7d096 100644 --- a/docs/source_docs/user_guide/inputs/advanced.rst +++ b/docs/source_docs/user_guide/inputs/advanced.rst @@ -37,12 +37,95 @@ GPU memory The following inputs must be preceded by "amrex." -+------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| | Description | Type | Default | -+========================+=======================================================================+=============+==============+ -| the _arena_is_managed | Abort if an invalid floating point exception is encountered. | Int | 0 | -+------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| abort_on_out_of_gpu_memory | rt if a division by zero is computed. | Int | 0 | -+------------------------+-----------------------------------------------------------------------+-------------+--------------+ -| the_area_init_size | Abort if and overflow is detected. | Int | 0 | -+------------------------+-----------------------------------------------------------------------+-------------+--------------+ ++----------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| | Description | Type | Default | ++============================+=======================================================================+=============+==============+ +| the_arena_is_managed | Abort if an invalid floating point exception is encountered. | Int | 0 | ++----------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| the_arena_init_size | Abort if and overflow is detected. | Int | 0 | ++----------------------------+-----------------------------------------------------------------------+-------------+--------------+ +| abort_on_out_of_gpu_memory | Abort if a division by zero is computed. | Int | 0 | ++----------------------------+-----------------------------------------------------------------------+-------------+--------------+ + + +Load balancing +-------------- + +Note that when running a granular simulation, i.e., no fluid phase, :cpp:`mfix.dual_grid` must be 0. Hence, +the :cpp:`particles.max_grid_size` (in each direction) have no meaning. Therefore the fluid grid and tile +sizes should be set for particle load balancing. It may also be necessary to set the blocking factors to 1. + + +The following inputs must be preceded by "mfix." and determine how we load balance: + ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| | Description | Type | Default | ++======================+=======================================================================+=============+==============+ +| dual_grid | If true then use the "dual_grid" approach for load balancing | Bool | False | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| regrid_int | How often to regrid (in number of steps at level 0) | Int | -1 | +| | if regrid_int = -1 then no regridding will occur | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| load_balance_fluid | Only relevant if (dual_grid); if so do we also regrid mesh data | Int | 0 | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| load_balance_type | What strategy to use for load balancing | String | KnapSack | +| | Options are "KnapSack", "SFC", or "Greedy" | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| knapsack_weight_type | What weighting function to use if using Knapsack load balancing | String | RunTimeCosts | +| | Options are "RunTimeCosts" or "NumParticles"" | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| knapsack_nmax | Maximum number of grids per MPI process if using knapsack algorithm | Int | 128 | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| greedy_dir | The direction in which the greedy algorithm cuts overloaded boxes | Int | 0 | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| greedy_min_grid_size | The minimum particle grid size in the greedy load balance algorithm | Int | 2 | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| greedy_3d | Partition particle grids in 3D with the greedy algorithhm | Bool | False | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| overload_tolerance* | The ratio between the maximum workload and the average workload | Real | 1.2 | +| | in the greedy algorithm | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| underload_tolerance* | The ratio between the minimum workload and the average workload | Real | 0.8 | +| | in the greedy algorithm | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| grid_pruning | Remove all covered grids from the base mesh; this may result in | Bool | False | +| | disjoined grids | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ + +\* The greedy partitioning algorithm uses the tolerances to set the expected +workload range for each rank but doesn't strictly enforce them. The algorithm +creates a partition as balance as possible even if the partition doesn't +meet the tolerances. + +To allow a user to verify the breakdown of fluid grids created before running a full simulation, an input option, +:cpp:`mfix.only_print_grid_report` is supported. By default, it is :cpp:`False`. When set to :cpp:`True`, the run uses +minimal memory to print the grid coverage report and exits immediately after that. + + +The following inputs are defined using the ``particles`` prefix. + ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| | Description | Type | Default | ++======================+=======================================================================+=============+==============+ +| max_grid_size_x | Maximum number of cells at level 0 in each grid in x-direction | Int | 32 | +| | for grids in the ParticleBoxArray if dual_grid is true | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| max_grid_size_y | Maximum number of cells at level 0 in each grid in y-direction | Int | 32 | +| | for grids in the ParticleBoxArray if dual_grid is true | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ +| max_grid_size_z | Maximum number of cells at level 0 in each grid in z-direction | Int | 32 | +| | for grids in the ParticleBoxArray if dual_grid is true. | | | ++----------------------+-----------------------------------------------------------------------+-------------+--------------+ + + +.. todo:: Is this needed anymore? + + The following inputs must be preceded by "particles." + + +----------------------+-----------------------------------------------------------------------+-------------+--------------+ + | | Description | Type | Default | + +======================+=======================================================================+=============+==============+ + | reduceGhostParticles | Remove unused ghost particles from communication, | Bool | True | + | | only supported by GPU build | | | + +----------------------+-----------------------------------------------------------------------+-------------+--------------+ + diff --git a/docs/source_docs/user_guide/inputs/domain.rst b/docs/source_docs/user_guide/inputs/domain.rst index f1a193318d463b4a05b3059efd4095832a83f68e..fe18fc18467de6f64f67cab8eafe61c76ac8d945 100644 --- a/docs/source_docs/user_guide/inputs/domain.rst +++ b/docs/source_docs/user_guide/inputs/domain.rst @@ -8,6 +8,8 @@ low and high corners of the cuboid are defined by the ``prob_lo`` and ``prob_hi` The following inputs are defined using the ``geometry`` prefix. +.. _InputsTable_domain: + +-----------------+-----------------------------------------------------------------------+-------------+-----------+ | | Description | Type | Default | +=================+=======================================================================+=============+===========+ diff --git a/docs/source_docs/user_guide/inputs/images/geometry/mesh_lev0_ex.png b/docs/source_docs/user_guide/inputs/images/geometry/mesh_lev0_ex.png new file mode 100644 index 0000000000000000000000000000000000000000..de1f7c1c1b9721cb93246729404b4271c294bc30 Binary files /dev/null and b/docs/source_docs/user_guide/inputs/images/geometry/mesh_lev0_ex.png differ diff --git a/docs/source_docs/user_guide/inputs/mesh_and_gridding.rst b/docs/source_docs/user_guide/inputs/mesh_and_gridding.rst index 96f6d3fc3f84dcaf65c4b59a647bdf9b47e5c5e9..82fbc73354c3301b2c1a83c50834cd20075757dd 100644 --- a/docs/source_docs/user_guide/inputs/mesh_and_gridding.rst +++ b/docs/source_docs/user_guide/inputs/mesh_and_gridding.rst @@ -1,84 +1,153 @@ .. sec:InputsMeshAndGridding: -Mesh and gridding -================= - -Mesh, grids and tiles ---------------------- +Mesh, grids, and tiles +====================== Mesh -^^^^ +---- + +.. rubric:: Level-0 mesh + +The following inputs are defined using the ``amr`` prefix. -There are some settings we can specify for the mesh and the automatic mesh -refinement algorithm. These settings must be preceded by "amr." in the input -file. +.. _InputsTable_mesh: +----------------------+-----------------------------------------------------------------------+-------------+-----------+ | | Description | Type | Default | +| | | | | +======================+=======================================================================+=============+===========+ -| max_level | Maximum level of refinement allowed (0 when single-level) | Int | 0 | +| max_level | Maximum level of refinement. The default value, 0, means that there | int | 0 | +| | is one cell size covering the entire domain (i.e., single-level). | | | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| n_cell | Number of cells at level 0 in each coordinate direction | ints | 0 0 0 | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ + +The level 0 mesh spacing is computed for each direction by dividing the :ref:`domain length` by the +number of cells. The mesh spacing is **required** to be the same in all directions: + +.. math:: + + \frac{\text{prob_hi[0] - prob_lo[0]}}{\text{n_cell[0]}} + = \frac{\text{prob_hi[1] - prob_lo[1]}}{\text{n_cell[1]}} + = \frac{\text{prob_hi[2] - prob_lo[2]}}{\text{n_cell[2]}} + + +The inputs for defining the mesh for a single-level simulation are demonstrated in the +:ref:`following example` and illustrated in :numref:`fig_basic_mesh_ex`. +In this example, the domain is a :math:`4 \times 1 \times 1` cuboid, and there are +:math:`32 \times 8 \times 8` cells in the *X*, *Y*, and *Z* directions, respectively. +The result is a uniform mesh spacing of :math:`0.125` *m* in all three directions. + +.. _inputs_mesh_ex: + +.. code-block:: bash + :caption: Snippet of intpus for mesh example. This is not a complete input file. + + # Define periodicity and domain extents + # ------------------------------------------------------------- + geometry.coord_sys = 0 # Cartesian coordinates + geometry.is_periodic = 0 0 0 # periodicity for each direction + geometry.prob_lo = 0. 0. 0 # lo corner of physical domain. + geometry.prob_hi = 4. 1. 1. # hi corner of physical domain + + # Define the maximum level of refinement and number of cells + # ------------------------------------------------------------- + arm.max_level = 0 + arm.n_cell = 32 8 8 + + +.. _fig_basic_mesh_ex: + +.. figure:: ./images/geometry/mesh_lev0_ex.png + :width: 100% + :align: center + :alt: domain used with box embedded boundary + + Example of a single-level mesh. + +.. warning:: + + MFIX-Exa simulations with a non-uniform mesh will not run. + + +----- + + +.. rubric:: Mesh refinement + + +The following inputs are defined using the ``amr`` prefix. These inputs control the automatic +mesh refinement algorithm and are only applicable when ``amr.max_level > 0``. + ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| | Description | Type | Default | ++======================+=======================================================================+=============+===========+ | grid_eff | Threshold value to ensure grids do not contain too large a fraction | Real | 0.7 | | | of un-tagged cells. | | | -| | Applicable only when mesh refinement is enabled ("max_level" > 0). | | | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| n_error_buf | Controls the number of tagged cells before grids are defined. Used | Int | 1 | +| n_error_buf | Controls the number of tagged cells before grids are defined. Used | int | 1 | | | to ensure coarse/fine boundaries are not too close to tagged cells. | | | -| | Applicable only when mesh refinement is enabled ("max_level" > 0). | | | -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| n_cell | Number of cells at level 0 in each coordinate direction | Ints | 0 0 0 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ +.. caution:: + + **Mesh refinement limitations:** + + * MFIX-Exa simulations with particles currently do not support mesh refinement. + +See the `AMReX documentation `_ +for details on the adaptive mesh refinement algorithms. + +----- + Grids -^^^^^ +----- -The following inputs must be preceded by "amr." and determine how we create the grids. +The following inputs are defined using the ``amr`` prefix. +----------------------+-----------------------------------------------------------------------+-------------+-----------+ | | Description | Type | Default | +======================+=======================================================================+=============+===========+ -| max_grid_size_x | Maximum number of cells at level 0 in each grid in x-direction | Int | 32 | +| max_grid_size_x | Maximum number of cells at level 0 in each grid in *X* | int | 32 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| max_grid_size_y | Maximum number of cells at level 0 in each grid in y-direction | Int | 32 | +| max_grid_size_y | Maximum number of cells at level 0 in each grid in *Y* | int | 32 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| max_grid_size_z | Maximum number of cells at level 0 in each grid in z-direction | Int | 32 | +| max_grid_size_z | Maximum number of cells at level 0 in each grid in *Z* | int | 32 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| blocking_factor_x | Each grid must be divisible by blocking_factor_x in x-direction | Int | 8 | +| blocking_factor_x | Each grid in *X* must be divisible by ``blocking_factor_x`` | int | 8 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| blocking_factor_y | Each grid must be divisible by blocking_factor_y in y-direction | Int | 8 | +| blocking_factor_y | Each grid in *Y* must be divisible by ``blocking_factor_y`` | int | 8 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| blocking_factor_z | Each grid must be divisible by blocking_factor_z in z-direction | Int | 8 | +| blocking_factor_z | Each grid in *Z* must be divisible by ``blocking_factor_z`` | int | 8 | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ +The domain is decomposed into *grids* by dividing the number of cells by the max grid size +for each direction (e.g., ``n_cells[0]/max_grid_size_x``). The blocking factor ensures that +the grids will be sufficiently coarsenable for good multigrid performance; therefore, the +``max_grid_size`` must be divisible by the corresponding ``blocking_factor``. -The following inputs must be preceded by "particles." +.. note:: + + The `AMReX documentation `_ contains a significant + amount of information on grid creation and load balancing. Users are **strongly** encouraged to + read the relevant sections. + + +----- -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| | Description | Type | Default | -+======================+=======================================================================+=============+==============+ -| max_grid_size_x | Maximum number of cells at level 0 in each grid in x-direction | Int | 32 | -| | for grids in the ParticleBoxArray if dual_grid is true | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| max_grid_size_y | Maximum number of cells at level 0 in each grid in y-direction | Int | 32 | -| | for grids in the ParticleBoxArray if dual_grid is true | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| max_grid_size_z | Maximum number of cells at level 0 in each grid in z-direction | Int | 32 | -| | for grids in the ParticleBoxArray if dual_grid is true. | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ Tiles -^^^^^ +----- -The following inputs must be preceded by "fabarray_mfiter." and determine how we create the logical tiles: +The following inputs are defined using the ``fabarray`` prefix. +----------------------+-----------------------------------------------------------------------+----------+-------------+ | | Description | Type | Default | +======================+=======================================================================+==========+=============+ -| tile_size | Maximum number of cells in each direction for (logical) tiles | IntVect | 1024000 | -| | (3D CPU-only) | | 1024000,8,8 | +| mfiter_tile_size | Maximum number of cells in each direction for (logical) tiles | IntVect | 1024000,8,8 | +----------------------+-----------------------------------------------------------------------+----------+-------------+ -The following inputs must be preceded by "particles." +The following inputs are defined using the ``particles`` prefix. +----------------------+-----------------------------------------------------------------------+-------------+--------------+ | | Description | Type | Default | @@ -87,69 +156,9 @@ The following inputs must be preceded by "particles." | | in the ParticleBoxArray if dual_grid is true. | | | +----------------------+-----------------------------------------------------------------------+-------------+--------------+ +When running on shared memory machines using an OpenMP enabled executable, *grids* are subdivided into +*tiles* and iterated over to improve data locality by cache blocking. -Load balancing --------------- - -Note that when running a granular simulation, i.e., no fluid phase, :cpp:`mfix.dual_grid` must be 0. Hence, -the :cpp:`particles.max_grid_size` (in each direction) have no meaning. Therefore the fluid grid and tile -sizes should be set for particle load balancing. It may also be necessary to set the blocking factors to 1. - - -The following inputs must be preceded by "mfix." and determine how we load balance: - -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| | Description | Type | Default | -+======================+=======================================================================+=============+==============+ -| dual_grid | If true then use the "dual_grid" approach for load balancing | Bool | False | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| regrid_int | How often to regrid (in number of steps at level 0) | Int | -1 | -| | if regrid_int = -1 then no regridding will occur | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| load_balance_fluid | Only relevant if (dual_grid); if so do we also regrid mesh data | Int | 0 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| load_balance_type | What strategy to use for load balancing | String | KnapSack | -| | Options are "KnapSack", "SFC", or "Greedy" | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| knapsack_weight_type | What weighting function to use if using Knapsack load balancing | String | RunTimeCosts | -| | Options are "RunTimeCosts" or "NumParticles"" | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| knapsack_nmax | Maximum number of grids per MPI process if using knapsack algorithm | Int | 128 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| greedy_dir | The direction in which the greedy algorithm cuts overloaded boxes | Int | 0 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| greedy_min_grid_size | The minimum particle grid size in the greedy load balance algorithm | Int | 2 | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| greedy_3d | Partition particle grids in 3D with the greedy algorithhm | Bool | False | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| overload_tolerance* | The ratio between the maximum workload and the average workload | Real | 1.2 | -| | in the greedy algorithm | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| underload_tolerance* | The ratio between the minimum workload and the average workload | Real | 0.8 | -| | in the greedy algorithm | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ -| grid_pruning | Remove all covered grids from the base mesh; this may result in | Bool | False | -| | disjoined grids | | | -+----------------------+-----------------------------------------------------------------------+-------------+--------------+ - -\* The greedy partitioning algorithm uses the tolerances to set the expected -workload range for each rank but doesn't strictly enforce them. The algorithm -creates a partition as balance as possible even if the partition doesn't -meet the tolerances. - -To allow a user to verify the breakdown of fluid grids created before running a full simulation, an input option, -:cpp:`mfix.only_print_grid_report` is supported. By default, it is :cpp:`False`. When set to :cpp:`True`, the run uses -minimal memory to print the grid coverage report and exits immediately after that. - - -.. todo:: Is this needed anymore? - - The following inputs must be preceded by "particles." - - +----------------------+-----------------------------------------------------------------------+-------------+--------------+ - | | Description | Type | Default | - +======================+=======================================================================+=============+==============+ - | reduceGhostParticles | Remove unused ghost particles from communication, | Bool | True | - | | only supported by GPU build | | | - +----------------------+-----------------------------------------------------------------------+-------------+--------------+ +.. note:: + MFIX-Exa disables tiling when using GPU accelerators.