Commit 0c41207d authored by Ann Almgren's avatar Ann Almgren
Browse files

update re dual grid

parent 9478e9b6
Loading
Loading
Loading
Loading
+9 −13
Original line number Diff line number Diff line
@@ -4,17 +4,13 @@
Gridding and Load Balancing
===========================

AMReX provides a great deal of generality when it comes to how to decompose the 
computational domain into individual logically rectangular grids, and how to distribute
those grids to MPI ranks.  We use the phrase "load balancing" here to refer to the combined process
of grid creation (and re-creation when regridding) and distribution of grids to MPI ranks.

Even for single-level calculations, AMReX provides the flexibility to have different size grids,
MFiX-Exa has a great deal of flexibility when it comes to how to decompose the 
computational domain into individual rectangular grids, and how to distribute
those grids to MPI ranks.  There can be grids of different sizes, 
more than one grid per MPI rank, and different strategies for distributing the grids to MPI ranks.

For multi-level calculations, the same principles for load balancing apply as in single-level calculations,
but there is additional complexity in how to tag cells for refinement and how to create the 
union of grids at levels > 0 where that union most likely does not cover the computational domain.
We use the phrase "load balancing" here to refer to the combined process
of grid creation (and re-creation when regridding) and distribution of grids to MPI ranks.

See :ref:`sec:grid_creation` for grids are created, i.e. how the :cpp:`BoxArray` on which 
:cpp:`MultiFabs` will be built is defined at each level.
@@ -23,10 +19,10 @@ See :ref:`sec:load_balancing` for the strategies AMReX supports for distributing
grids to MPI ranks, i.e. defining the :cpp:`DistributionMapping` with which 
:cpp:`MultiFabs` at that level will be built.  

We also note that we can create separate grids, and map them in different ways to MPI ranks, for 
different types of data in a single calculation.  We refer to this as the "dual grid approach"
and the most common usage is to load balance mesh and particle data separately. See :ref:`sec:dual_grid`
for more about this approach.
MFiX-Exa also allows for the "dual grid approach", in which mesh and particle data are allocated
on different box layouts with different mappings to MPI ranks.  This option is enabled 
by setting :cpp:`amr.dual_grid = 1` in the inputs file.
See :ref:`sec:dual_grid` for more about this approach.

When running on multicore machines with OpenMP, we can also control the distribution of 
work by setting the size of grid tiles (by defining :cpp:`fabarray_mfiter.tile_size`), and if relevant, of