Update gridding

.. role:: fortran(code)

   :language: fortran

.. _ss:grid_creation:

.. _sec:grid_creation:

Grid Creation

-------------

To run MFiX-Exa you must specifiy the domain size by specifying :cpp:`n_cell` -- 

this is the number of cells spanning the domain in each coordinate direction at the coarsest level (level 0).

To run MFiX-Exa you must specify :cpp:`n_cell` in the inputs file -- 

this is the number of cells spanning the domain in each coordinate direction at level 0.

Users often specify :cpp:`max_grid_size` as well. The default load balancing algorithm then divides the 

domain in every direction so that each grid is no longer than :cpp:`max_grid_size` in that direction.

If not specified by the user, :cpp:`max_grid_size` defaults to 32 in 3D (in each coordinate direction).

If not specified by the user, :cpp:`max_grid_size` defaults to 128 in 2D and 32 in 3D (in each coordinate direction).

Another popular input is :cpp:`blocking_factor`.  The value of :cpp:`blocking_factor` 

constrains grid creation in that in that each grid must be divisible by :cpp:`blocking_factor`.  

   (or :cpp:`blocking_factor_x`, :cpp:`blocking_factor_y` and :cpp:`blocking_factor_z`) must be used.  

   If you don't specify as many integers as there are levels, the final value will be used for the remaining levels.

Additional notes:

 - to create identical grids of a specific size, e.g. of length *m* in each direction, 

   then set :cpp:`max_grid_size` = *m* and :cpp:`blocking_factor` = *m*.

 - note that :cpp:`max_grid_size` is just an upper bound; with :cpp:`n_cell = 48` 

   and :cpp:`max_grid_size = 32`, we will typically have one grid of length 32 and one of length 16.

The grid creation proceeds as follows:

The grid creation process at level 0 proceeds as follows (if not using the KD-tree approach):

#. The domain is initially defined by a single grid of size :cpp:`n_cell`.

   at this level, then the grids at this level are further divided until Ngrids >= Nprocs

   (unless doing so would violate the :cpp:`blocking_factor` criterion).

#. The creation of grids at higher levels begins by tagging cells at the coarser level and follows

   the Berger-Rigoutsis clustering algorithm with the additional constraint of satisfying

   the :cpp:`blocking_factor` criterion.

.. role:: cpp(code)

   :language: c++

.. role:: fortran(code)

   :language: fortran

.. _sec:load_balancing:

Load Balancing

--------------

The process of load balancing is typically independent of the process of grid creation; 

the inputs to load balancing are a given set of grids with a set of weights 

assigned to each grid.  (The exception to this is the KD-tree approach in which the

grid creation process is governed by trying to balance the work in each grid.)

Single-level load balancing algorithms are sequentially applied to each AMR level independently, 

and the resulting distributions are mapped onto the ranks taking into account the weights 

already assigned to them (assign heaviest set of grids to the least loaded rank)

Options supported by AMReX include:

- Knapsack: the default weight of a grid in the knapsack algorithm is the number of grid cells, 

  but AMReX supports the option to pass an array of weights – one per grid – or alternatively 

  to pass in a MultiFab of weights per cell which is used to compute the weight per grid

- SFC: enumerate grids with a space-filling Z-morton curve, then partition the 

  resulting ordering across ranks in a way that balances the load

- Round-robin: sort grids and assign them to ranks in round-robin fashion -- specifically

  FAB *i* is owned by CPU *i*%N where N is the total number of MPI ranks.

.. role:: cpp(code)

   :language: c++

.. _Chap:Managing the Grid Hierarchy:

Managing the Grid Hierarchy

Gridding and Load Balancing

===========================

Computational load balancing is based on several different steps:

AMReX provides a great deal of generality when it comes to how to decompose the 

computational domain into individual logically rectangular grids, and how to distribute

those grids to MPI ranks.  We use the phrase "load balancing" here to refer to the combined process

of grid creation (and re-creation when regridding) and distribution of grids to MPI ranks.

Even for single-level calculations, AMReX provides the flexibility to have different size grids,

more than one grid per MPI rank, and different strategies for distributing the grids to MPI ranks.

For multi-level calculations, the same principles for load balancing apply as in single-level calculations,

but there is additional complexity in how to tag cells for refinement and how to create the 

union of grids at levels > 0 where that union most likely does not cover the computational domain.

See :ref:`sec:grid_creation` for grids are created, i.e. how the :cpp:`BoxArray` on which 

:cpp:`MultiFabs` will be built is defined at each level.

#. The first component of load balancing is the creation of individual grids, 

   i.e. defining the :cpp:`BoxArray` on which :cpp:`MultiFabs` will be built at each level 

See :ref:`sec:load_balancing` for the strategies AMReX supports for distributing

grids to MPI ranks, i.e. defining the :cpp:`DistributionMapping` with which 

:cpp:`MultiFabs` at that level will be built.  

#. The second component is distribution of these grids to MPI processes, 

   i.e. defining the :cpp:`DistributionMapping` with which :cpp:`MultiFabs` at that level will be built.

   How we do this depends on what strategy we specify and what work estimate we use 

We also note that we can create separate grids, and map them in different ways to MPI ranks, for 

different types of data in a single calculation.  We refer to this as the "dual grid approach"

and the most common usage is to load balance mesh and particle data separately. See :ref:`sec:dual_grid`

for more about this approach.

We note that we can load balance the mesh and particle data separately; see :ref:`dual_grid`

When running on multicore machines with OpenMP, we can also control the distribution of 

work by setting the size of grid tiles (by defining :cpp:`fabarray_mfiter.tile_size`), and if relevant, of 

particle tiles (by defining :cpp:`particle.tile_size`).  We can also specify the strategy for assigning 

tiles to OpenMP threads.  See :ref:`sec:basics:mfiter:tiling:` for more about tiling.

.. toctree::

   :maxdepth: 1

   GridCreation

   DualGrid

   LoadBalancing