Anyone knows why my DMP simulation is interrupted as shown in the screenshot?
You could be out of memory.
May be… I have checked the memory usage during a simulation and I have found out that the memory usage increases over time. During the first times steps it was about 7 GB, 3 hrs later it was 15 GB and after 8 hrs it is now about 49 GB… WHY???
@mark.meredith @onlyjus As I have attached the screenshots, you can see that the memory usage was increased over time and the code now uses almost all the RAM!!!Stopping the simulation would rapidly drop the memory usage and running it again leads to a same situation. Our lab member believes that it is a memory leak, what do you think?Is it a code bug???
@mark.meredith @onlyjuscone-particle-1.mfx (14.2 KB)
It could be. You are not feeding particles correct?
What versions of the compiler and mpi are you using?
Thanks for your response, Justin. I am not sure what you mean by not feeding particles correct!? I have the same problem when running the 3D Cylindrical FB of the tutorial.
Also, My compiler is GNU Fortran 5.5.0 and mpi version is 2.1.1.
What do you think I should do?
Also, I built the DMP solver and run it using the following command. The build was successful!
build_mfixsolver -DCMAKE_BUILD_TYPE=Debug --batch --dmp -j -DMPI_Fortran_COMPILER=mpifort -DCMAKE_Fortarn_FLAGS="-o3 -g -fcheck=all"
and
mpirun -np 8 ./mfixsolver -f Cubic.mfx NODESi=2 NODESj=2 NODESk=2
5.5.0 is pretty old. I usually us 8.2. Can you try a newer compiler?
Thanks @onlyjus. I will try that and let you know!
BTW, If I want to use my own machine, having a single CPU with 8 local cores, should I use SMP by setting NODESi, NODESJ, and NODESK equal to 1 in mfx file and using the following commands:
build_mfixsolver -DCMAKE_BUILD_TYPE=Debug --batch --smp -j -SMP_Fortran_COMPILER=gfortran -DCMAKE_Fortarn_FLAGS="-o3 -g -fcheck=all"
and
env OMP_NUM_THREADS=8 ./mfixsolver -f FILENAME.mfx
or I can also use the DMP solver by setting the number of available local cores equal to the spatial domain decomposition??? (DMP also uses all available cores on the single CPU when I check the Ubuntu system monitor )
You can use either DMP or SMP or both. I don’t have much experience with SMP mainly because I run simulations across multiple nodes on our HPC, so I have to use DMP.
@onlyjus I address the issue. The problem is that using dmp on a single CPU with local cores apparently uses the number of cores that you need but it will lead to the memory leak, but using SMP does not leas to the memory leak. @mark.meredith may tell us more about this?