Hello everyone,
(1) Currently I add some code through CALC_FORCE_DEM and CALC_COLLISION_WALL files, if I want to write it as DMP code what should I do to try?
(2) What is the difference between DEM parallelism and fluid parallelism?All I know is that information is exchanged through ghost grids when dealing with performing fluid cell parallel calculations.
Hello everyone!
Can you give me some guidance or what I should work on? I’m very confused right now.
- It depends what you mean by “some code”. If you are using the same dem particle arrays and only need local variables for your new calculations, you don’t need anything else. If you have new dem particles arrays, then you will need to exchange data as particles are near or cross process boundaries (see below).
- With DEM, you need to exchange data similar to the fluid cell ghost cells, except we have ghost particles. If you create new particle arrays, you will need to add then when you pack and unpack the data (see
mpi_pack_des_mod.fandmpi_unpack_des_mod.f)
Thanks Jeff, I’ll try it.
Is this problem solved, please
@jeff.dietiker Hi Jeff, sorry for the late reply. I would like to know if a particle in a process can access all other particles in DMP.
Sorry Li, I don’t quite understand yet.
Without data exchange (Message Passing), particles that belong to a process can only access particles on the same process. If you need to exchange data between processes, you need to use some MPI calls. This is a specific programming paradigm that takes time to learn.
Yes, Jeff, I have learned some knowledge about MPI and also reviewed the mpi_pack_des_mod.f and mpi_unpack_des_mod.f files. As I understand it, particles in the current process can receive information about particles at the neighbourhood boundary. However, I am unclear about the following concept:
(1) When packing ghost particles (SUBROUTINE DESMPI_PACK_GHOSTPAR(PFACE)), what is the range of the ghost particles? Is it a single layer of the grid at the boundary of the current process?
(2) Is there a way to obtain information about all other particles?
- Yes the range is a ghost layer near the common boundary between processes.
- You can manually pack, send and receive information about other particles, but you will need to develop your own MPI calls. You need to be careful with the amount of data you send/receive to minimize communication overhead.
Thank you for your response, Jeff. I am currently attempting to minimise communication costs. I would like to narrow the calculation range to the neighbours of each particle. However, I need to determine under what conditions particle A can become a neighbour of particle B, or within what range particle B’s neighbours are located.
I am not sure what you want to compute, but if you take a look at mode/des/calc_force.f, you will see we store the neighbor information in NEIGHBOR_INDEX and NEIGBHORS. Maybe this is a good starting point if you want to get more familiar with the code.
Okay, Jeff, the main file I modified is calc_force_dem.f. I have a question. When looping through neighbouring particles I in the file calc_force_dem.f, if neighbouring particle I is a ghost particle in the main particle LL’s process, will the value of particle I (such as FC, Q_SOURCE, etc.) be changed? I don’t think so, but MFIX does not perform any special operations on the following code.
@jeff.dietiker I believe this is a critical issue. When I use the DES_USR_VAR test, I increment DES_USR_VAR(1,LL) continuously and decrement DES_USR_VAR(1,I) continuously. When I sum DES_USR_VAR1 in Paraview, the correct result should be 0, but it actually keeps increasing. However, when I use IF(IS_NORMAL(LL).and.IS_NORMAL(I)) for control, I can obtain the correct result.
I think this must be an MPI communication issue. However, there is no special handling for FC and Q_source in the official code, so does this issue still exist?
@jeff.dietiker Hi Jeff, I’m sorry to bother you. But this is indeed a tricky problem for me. Compared to previous versions, this part of the code has not changed, so I’m not sure if it’s a problem with the official code or something else.
@jeff.dietiker Hi Jeff, sorry to bother you again. I have been exploring for over a week, but still have not found a solution. I kindly request that you could tell me the reason.
Hi @Ethan_MFIX,
Sorry for the late reply. Not sure if you’ve already figured this out, but the MPI for des_usr_var is actually working properly. In MFIX, we handle MPI communication through the exchange of normal and ghost particle information. It’s tracked using global particle IDs, though each processor maintains its own local IDs.
Please see this illustration, in the DMP version,
LL and I in the code refer to local particle IDs, not global particle IDs. So In this two-particle system, each processor handles only one normal particle:For processor 1, local ID
LL = 1 corresponds to global particle ID 1 (the left sphere), which is the normal particle on processor 1. Then you can find I = 2 refers to global particle ID 2, which is a ghost particle on this processor.For processor 2, local ID
LL = 1 corresponds to global particle ID 2 (the right sphere), which is the normal particle on processor 2. I = 2 now refers to the left particle, which is a ghost particle on this processor.In MFIX, inter-processor communication relies on normal and ghost particles. Therefore, simply doing
des_usr_var(1, LL) = des_usr_var(1, LL) + 1
des_usr_var(1, I) = des_usr_var(1, I) - 1
in calc_force_dem.f will not result in a net zero sum, again, LL and I here are local IDs. This is because each processor only updates normal particles during MPI communication. The values of ghost particle (I) are overwritten by MPI from the corresponding normal particle on the owning processor.
That means, if you run 10 solid time steps, you’ll see in ParaView that each particle will have des_usr_var(1,:) = 10, as the increments accumulate only on the normal particles.
You will only observe the sum becoming zero if both particles are on the same processor and are normal particles (i.e., no ghosting involved).
1 Like
Hi @kerenjie127 .Thank you very much for your answer. I understand how the processor handles normal particles and ghost particles. My understanding of neighbour traversal is as follows: if ll’s neighbours are i and j, and i’s neighbours are ll and j, then during neighbour traversal, ll will traverse i and j, but i will not traverse ll. Therefore, in process 2, des_usr_var(1, I) is not updated. This is my understanding, but it may be incorrect.
Yeah, that’s true for a single rank. But in DMP, it’s a different story due to the definition of global and local IDs.
Let’s say ll, i, and j are the local IDs on process 1. You might have values like ll = 1, i = 2, and j = 3, which refer to global IDs 1, 2, and 3 respectively. ll could be normal particle, and i and j are ghost particles, it will follow the traverse logic you mentioned.
However, on process 2, you also have ll = 1, i = 2, j = 3, but they refer to global IDs 2, 3, and 1. Here, ll will traverse i and j as well. Process 2 will update both des_usr_var(1,ll) and des_usr_var(1,i) and send those values to process 1 during MPI exchange.
So, if you look back at process 1:
des_usr_var(1,ll)is updated by process 1 itself,des_usr_var(1,i)anddes_usr_var(1,j)receive values from process 2.
Similar thing happens on process 2. As a result, all values are updated properly across processes.