I ran on a cluster a 3D polydisperse DEM simulation with over 2 million particles (with air as a fluid), and when doing a restart_1, I have the following error:
Fatal error in PMPI_Wait: Message truncated, error stack:
PMPI_Wait(219)…: MPI_Wait(request=0x47a60750, status=0x3c32550) failed
MPIR_Wait_impl(100)…: fail failed
MPID_nem_gen2_module_iStartContigMsg(438): Message from rank 180 and tag 5 truncated; 46739176 bytes received but buffer size is 46098720
I have never had an issue like this. Has anyone an idea what is wrong here? (Restart_2 also does not work)
Thanks