Some questions about DMP

MFiX
#1

do ijk = 1, dimension_3
if (fluid_at(ijk)) then…
end do

do ijk = ijkstart3, ijkend3
if(fluid_at(ijk)) then
end do

I would like to know what is the difference between the two? Can ”fluid_at(ijk)“ prevent traversing to a ghost unit that is not that rank in the loop ”do ijk = 1, dimension_3“?

1 Like
#2

Where did you find these code examples?

#3

I’ve been writing some code lately, but I’m not very familiar with DMP. I would like to ask you some questions in order to prevent the code from going wrong when using DMP.

#4

Sure, go ahead and ask your questions. I wanted to know where the two loops above came from - did you find those in MFiX source code? If so, where did they occur?

#5

Question1：
do ijk = ijkstart3, ijkend3
if(fluid_at(ijk)) then
…
endif
end do
This loop structure is common in mfix. I would like to know if ijkstart3 and ijkend3 in a rank contain ghost cells of the boundary ?

Question2：
do ijk = 1, dimension_3
if (fluid_at(ijk)) then
…
endif
end do
What happens if I use this structure in DMP? Will one rank change the values of other ranks responsible for grid cells in this loop?

Question 3:
What happens if I use the statement a( : )=b( : ) in DMP? Where the dimension size of a and b is DIMENSION_3.

Question 4:
What does send_recv do in MFIX? For example, “call send_recv(EP_G,2)”. What does the “2” mean?

Thank you very much for your patient answer!

#6

  1. Yes it contain ghost cells, in MFIX, we use “3” to denote two ghost layers.

  2. It really depends on fluid_at(ijk), each rank will check if this ijk cell has fluid.

  3. Each rank will copy values of b to a, each rank has their own a and b.

  4. send_recv is the mpi rank to rank communication, we wrapped several functions into a subroutine for convenient use. 2 is the number of layers in send and recv

#7

Thank you very much for your patience!
For questions 1 and 3:
I would like to know that if I change the cell value of level 2 in level 1, at this point if I use send_recv(), will the cell value of level 2 be changed?Or do rank 1 and rank 2 only exchange the values of the cells they are responsible for?
For questions 4：
What does the number of layers mean?

#8

you probably know about normal cells and ghost cells in the CFD parallelization. So ghost cells are 1 or 2 layers of grid cell in the subdomain boundaries.
For example, rank 1 has ghost cells, which are normal cells in rank 2. So once rank2 updates info about those normal cells, it will send those info to rank 1 by this send_recv() call. so rank 1 can use the boundary information of adjacent regions during local computations.

number of layers means the number of ghost cell layers. The number of ghost layers is based on the numerical method. 2 layers mean you are using second order scheme.

#9

Thanks again for your patient response!
I see what you mean. What does localized computation mean here? When I call send_recv(), can’t the ghost cell at rank 1 always store the exchanged values?

#10

a computational domain is distributed to different ranks, each rank is responsible for part of the domain (subdomain), so if we don’t do info exchange, a rank does not know what’s going on in other ranks. So this kind of computation is local. They don’t exchange until we ask them to do that.

ghost cells at rank 1 does store values sent by rank 2, since they are copies of normal cells in rank2, when rank 2 updates values, it need to tell rank 1 to update values in ghost cells. rank 1 does not calculate values for ghost cells by itself, it only receives values from rank 2.

1 Like
#11

Thanks, Renjie! I believe I fully understand your explanation now.