Re-indexing of mesh for 3D cylinder, conical bottom

MFiX
#1

Hi,
For a TFM bubbling fluidized bed (in a circular tank) I have turned on re-indexing, which I have a hunch can speed up simulations when running in parallel. Though I cannot find anything about this in the documentation. To me it seems that (dead) cells are removed. Are these then rearranged, meaning, should I try to improve the partition (i.e make it coarser) so that these parts do not become too small compared to the rest?

  RE-INDEXING CELLS FOR CARTESIAN GRID...
  Re-indexing: INFO: USE_DOLOOP was set to .TRUE.
  Re-indexing: Identifying dead cells ...
  Re-indexing: Arranging all interior cells in next contiguous block...
  Re-indexing: Shifting arrays...
  Re-indexing: Minimizing send and receive arrays for Send layer 1...
  Re-indexing: Minimizing send and receive arrays for Send layer 2...
  Re-indexing: Minimizing send and receive arrays for Receive layer 1...
  Re-indexing: Minimizing send and receive arrays for Receive layer 2...
  Re-indexing: Re-assigning cell classes...
  Re-indexing: New number of classes =          646
  Re-indexing: Writing IJK value in files...
 =============================================================================
     PROCESSOR    I-SIZE     J-SIZE     K-SIZE    # CELLS    # CELLS   DIFF.
                                                  (BCKGRD) (RE-INDEXED) (%)
 =============================================================================
          0         13         44         13      13872       5770    -58.4
          1         13         44         13      13872      11393    -17.9
          2         13         44         13      13872      11393    -17.9
          3         13         44         13      13872       5658    -59.2
          4         13         44         13      13872       6936    -50.0
          5         13         44         13      13872      13342     -3.8
          6         13         44         13      13872      13330     -3.9
          7         13         44         13      13872       6810    -50.9
          8         13         44         13      13872      11393    -17.9
          9         13         44         13      13872      13219     -4.7
         10         13         44         13      13872      13219     -4.7
         11         13         44         13      13872      11278    -18.7
         12         13         44         13      13872      13330     -3.9
         13         13         44         13      13872      13872      0.0
         14         13         44         13      13872      13872      0.0
         15         13         44         13      13872      13216     -4.7
         16         13         44         13      13872      11393    -17.9
         17         13         44         13      13872      13219     -4.7
         18         13         44         13      13872      13219     -4.7
         19         13         44         13      13872      11128    -19.8
         20         13         44         13      13872      13342     -3.8
         21         13         44         13      13872      13872      0.0
         22         13         44         13      13872      13872      0.0
         23         13         44         13      13872      13028     -6.1
         24         13         44         13      13872       5658    -59.2
         25         13         44         13      13872      11278    -18.7
         26         13         44         13      13872      11128    -19.8
         27         13         44         13      13872       5281    -61.9
         28         13         44         13      13872       6822    -50.8
         29         13         44         13      13872      13216     -4.7
         30         13         44         13      13872      13028     -6.1
         31         13         44         13      13872       6360    -54.2
 =============================================================================
 MAX # OF CELLS (BACKGRD)    =    13872     AT PROCESSOR:        0
 MAX # OF CELLS (RE-INDEXED) =    13872     AT PROCESSOR:       13
 DIFFERENCE (%)              =      0.0
 =============================================================================
#2

My guess is you won’t get much gain with re-indexing in your case because of the cylindrical geometry. Re-indexing helps if you have many dead cells in serial (say 50% dead cells or more). In parallel, you will get some speedup if all processes are able to reduce their active cell count. The bottleneck is the process with the most cells. Even if many processes can reduce their cell count, they will be waiting for the one with the most cells.

#3

Thanks, this was what I expected. Would you say it then makes sense to instead partition the mesh vertically, with only 2x2 in the horizontal plane (meaning symmetric partitions), and 4 or 8 vertically? Or would this be a bad idea if most of the solids are in the bottom partition “layer”, meaning this region is more computationally expensive than the rest, and so still being a bottleneck?

#4

For TFM, you can try a 2x8x2 partition so each partition has can reduce their cell count. Again with a mostly cylindrical geometry, it may not be much, but worth trying. If it was DEM, you would have much more of a load imbalance because the processors handling the top of the domain would be waiting for the processors handling the bottom.