From 789dd33897c995dc57029b0c50ed6f57cd168a3d Mon Sep 17 00:00:00 2001 From: Deepak Rangarajan Date: Thu, 31 Oct 2024 10:32:24 -0400 Subject: [PATCH 1/4] new options for MPMD --- docs/source_docs/user_guide/inputs/output.rst | 1 + .../user_guide/inputs/output/mpmd.rst | 24 +++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 docs/source_docs/user_guide/inputs/output/mpmd.rst diff --git a/docs/source_docs/user_guide/inputs/output.rst b/docs/source_docs/user_guide/inputs/output.rst index 4b9bad4..7ef1612 100644 --- a/docs/source_docs/user_guide/inputs/output.rst +++ b/docs/source_docs/user_guide/inputs/output.rst @@ -11,3 +11,4 @@ Stuff here? output/monitors output/ascent output/spatial_avgs + output/mpmd diff --git a/docs/source_docs/user_guide/inputs/output/mpmd.rst b/docs/source_docs/user_guide/inputs/output/mpmd.rst new file mode 100644 index 0000000..97ff21a --- /dev/null +++ b/docs/source_docs/user_guide/inputs/output/mpmd.rst @@ -0,0 +1,24 @@ +Multiple Program Multiple Data (MPMD) +-------------------------------------- + +MFIX-Exa utilizes the AMReX-MPMD interface to send data across to another program or application. In order to enable this feature, the executable has to be built with ``-DMFIX_MPMD = yes``. + +The following inputs must be preceded by "mfix." and control frequency and the data that will be sent to the other program. + ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| | Description | Type | Default | ++======================+=======================================================================+=============+===========+ +| mpmd_int | Frequency at which data is sent to the other application | Int | -1 | +| | when using MPMD. | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_per_approx | Time period for sending data to the other pipeline (approximate) | Real | -1 | +| | when using MPMD. | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_static_mfs | A list of the names of static multifabs that will be sent | String | None | +| | only once when using MPMD. Possible values are | | | +| | "volfrac" and "centroid". | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_mfs | A list of the names of multifabs that will be sent per set frequency | String | None | +| | when using MPMD. Possible values are "ep_g", "vel_g", "T_g" | | | +| | and "X_gk". | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ -- GitLab From 46f403ea1aec9aac9eb83825b88de99d540fceec Mon Sep 17 00:00:00 2001 From: Deepak Rangarajan Date: Thu, 2 Jan 2025 13:26:53 -0500 Subject: [PATCH 2/4] add python script description and limitations --- .../user_guide/inputs/output/mpmd.rst | 202 +++++++++++++++++- 1 file changed, 200 insertions(+), 2 deletions(-) diff --git a/docs/source_docs/user_guide/inputs/output/mpmd.rst b/docs/source_docs/user_guide/inputs/output/mpmd.rst index 97ff21a..6f97bf0 100644 --- a/docs/source_docs/user_guide/inputs/output/mpmd.rst +++ b/docs/source_docs/user_guide/inputs/output/mpmd.rst @@ -1,7 +1,10 @@ Multiple Program Multiple Data (MPMD) --------------------------------------- +===================================== -MFIX-Exa utilizes the AMReX-MPMD interface to send data across to another program or application. In order to enable this feature, the executable has to be built with ``-DMFIX_MPMD = yes``. +MFIX-Exa utilizes the AMReX-MPMD interface to send data across to another program or application. In order to enable this feature, the executable has to be built with ``-DMFIX_MPMD = yes``. + +Inputs +------ The following inputs must be preceded by "mfix." and control frequency and the data that will be sent to the other program. @@ -22,3 +25,198 @@ The following inputs must be preceded by "mfix." and control frequency and the d | | when using MPMD. Possible values are "ep_g", "vel_g", "T_g" | | | | | and "X_gk". | | | +----------------------+-----------------------------------------------------------------------+-------------+-----------+ + +Sample Python Program +---------------------- + +A sample python script that gathers and plots velocity statistics for the case of fluid flow through a pipe can be found +in ``tutorials/mpmd/main.py``. The script can be divided into the following sections: + +Initialize +~~~~~~~~~~ + +* Initialize AMReX::MPMD and leverage MPI from ``mpi4py`` to perform communication split. + + .. code-block:: python + + amr.MPMD_Initialize_without_split([]) + app_comm = MPI.COMM_WORLD.Split(amr.MPMD_AppNum(), amr.MPMD_MyProc()) + app_world_size = app_comm.Get_size() + app_rank = app_comm.Get_rank() + amr.initialize_when_MPMD([], app_comm) + +* Determine the C++ app's root process + + .. code-block:: python + + if app_rank == 0: + if amr.MPMD_MyProc() == app_rank: # first program + other_root = app_comm.Get_size() + print(f'other_root = {other_root}') + +* Create an MPMD::Copier object that gets the BoxArray information from the C++ app. + + .. code-block:: python + + copr = amr.MPMD_Copier(True) + + +Receive Once +~~~~~~~~~~~~ + +* Receive the ``Header`` information as a json string on the python root from the C++ root + and broadcast to all python ranks. + + .. code-block:: python + + header_json = "" + + if app_rank == 0: + buf = bytearray(10000) # Create a buffer to receive the message + MPI.COMM_WORLD.Recv([buf, MPI.CHAR], source=other_root) + header_json = buf.decode().strip('\x00') # Decode and strip null characters + + header_json = app_comm.bcast(header_json, root=0) + +* Receive all the static ``Multifab`` data. + + .. code-block:: python + + my_static_data = MyData() + for mf in header["data"]["static_mfs"]: + my_static_data.define_mf(copr, mf["n"], mf["c"]) + my_static_data.copy_mf(copr, mf["n"], mf["c"]) + + +Receive Until *End* +~~~~~~~~~~~~~~~~~~~ + +* Receive ``End Flag`` on the python root from the C++ root and broadbast to all python + ranks. If the flag is ``1``, break out of the loop. + + .. code-block:: python + + if app_rank == 0: + int_flags = np.empty(len(header["data"]["int_flags_root"]), dtype='i') + MPI.COMM_WORLD.Recv(int_flags, source=other_root) + print(f"app_rank = {app_rank}, int_flags = {int_flags})") + end = int_flags[0] + + end = app_comm.bcast(end, root=0) + + if end == 1: + break + +* Receive ``Reals`` on the python root from the C++ root and broadbast to all python + ranks. Save ``time`` to an array on the python root for plotting. + + .. code-block:: python + + if app_rank == 0: + reals = np.empty(len(header["data"]["reals_root"]), dtype=np.double) + MPI.COMM_WORLD.Recv(reals, source=other_root) + print(f"app_rank = {app_rank}, reals = {reals})") + time = reals[0] + time_arr.append(time) + + time = app_comm.bcast(time, root=0) + +* Receive ``MultiFab`` data and store to arrays on the python root as necessary. + In this example, the ``centerline`` u-velocity and the data needed to compute the + mean and variance of u-velocity on the central ``y-plane`` are stored as an array + in time. + + .. code-block:: python + + for mf in header["data"]["mfs"]: + my_data.copy_mf(copr, mf["n"], mf["c"]) + + for mfi in my_data.mfs["vel_g"]: + bx = mfi.validbox() + y_intrst_exists = True + z_intrst_exists = True + + if (j_intrst < bx.small_end[1] or j_intrst > bx.big_end[1]): + y_intrst_exists = False + + if (k_intrst < bx.small_end[2] or k_intrst > bx.big_end[2]): + z_intrst_exists = False + + if (not((y_intrst_exists or z_intrst_exists))): + continue + + ###.............. + + if ( y_intrst_exists and z_intrst_exists ): + np_array = np.array(vel_g_array[0,k_intrst,j_intrst,:]) + u_centerline[bx.small_end[0]:bx.small_end[0] + np_array.size] = np_array + + if (y_intrst_exists): + + ###.............. + + y_pl_npts += np.sum(y_volfrac_array) + y_pl_u_mn += np.sum(y_vel_g_array[0,:,:]) + y_pl_u2_mn += np.sum(y_vel_g_array[0,:,:]*y_vel_g_array[0,:,:]) + + # Reduce from all python ranks + y_pl_npts = app_comm.reduce(y_pl_npts,op=MPI.SUM,root=0) + y_pl_u_mn = app_comm.reduce(y_pl_u_mn,op=MPI.SUM,root=0) + y_pl_u2_mn = app_comm.reduce(y_pl_u2_mn,op=MPI.SUM,root=0) + u_centerline = app_comm.reduce(u_centerline,op=MPI.SUM,root=0) + + #................... + + if app_rank == 0: + y_pl_u_mn /= y_pl_npts + y_pl_u2_mn /= y_pl_npts + + y_pl_u_mn_arr.append(y_pl_u_mn) + y_pl_u_var_arr.append(y_pl_u2_mn-y_pl_u_mn*y_pl_u_mn) + u_centerline_arr.append(u_centerline) + + app_comm.barrier() + + +Plot +~~~~ + +* Plot figures on the python root using the collected arrays. + + .. code-block:: python + + if app_rank == 0: + fig, (ax1, ax2, ax3) = plt.subplots(1,3) + + ax1.plot(time_arr, y_pl_u_mn_arr) + ax1.set_title('mean') + ax1.set_xlabel('Time (s)') + + ax2.plot(time_arr, y_pl_u_var_arr) + ax2.set_title('var') + ax2.set_xlabel('Time (s)') + + ax3.plot(range(xlen), np.array(u_centerline_arr).mean(axis=0)) + ax3.set_title('centerline U (m/s)') + ax3.set_xlabel('i') + + plt.savefig('my_plot.png') + + +Finalize +~~~~~~~~ + +* Finalize AMReX and AMReX::MPMD. + + .. code-block:: python + + amr.finalize() + amr.MPMD_Finalize() + + + +Limitations +------------ + +* Does not work with simulations that require restarts. +* Cannot be used to send and receive particle data. -- GitLab From 10b9d9bbb0f2f62ea0d21a1b069bc6185d6a34d0 Mon Sep 17 00:00:00 2001 From: Jordan Musser Date: Tue, 7 Jan 2025 11:29:30 -0500 Subject: [PATCH 3/4] Split MPMD into inputs / refs sections --- docs/source_docs/index.rst | 1 + .../inputs/output => references}/mpmd.rst | 44 +++++-------------- docs/source_docs/user_guide/inputs/mpmd.rst | 37 ++++++++++++++++ docs/source_docs/user_guide/inputs/output.rst | 1 - .../user_guide/run-time_inputs.rst | 1 + 5 files changed, 49 insertions(+), 35 deletions(-) rename docs/source_docs/{user_guide/inputs/output => references}/mpmd.rst (68%) create mode 100644 docs/source_docs/user_guide/inputs/mpmd.rst diff --git a/docs/source_docs/index.rst b/docs/source_docs/index.rst index b5be310..ba50691 100644 --- a/docs/source_docs/index.rst +++ b/docs/source_docs/index.rst @@ -60,6 +60,7 @@ To learn more about the implementation, follow the following reference sections: references/Particles_Chapter references/EB_Chapter references/hpc + references/mpmd references/Debugging works_cited diff --git a/docs/source_docs/user_guide/inputs/output/mpmd.rst b/docs/source_docs/references/mpmd.rst similarity index 68% rename from docs/source_docs/user_guide/inputs/output/mpmd.rst rename to docs/source_docs/references/mpmd.rst index 6f97bf0..b27ac3d 100644 --- a/docs/source_docs/user_guide/inputs/output/mpmd.rst +++ b/docs/source_docs/references/mpmd.rst @@ -1,30 +1,14 @@ +.. _ReferenceMPMD: + Multiple Program Multiple Data (MPMD) ===================================== -MFIX-Exa utilizes the AMReX-MPMD interface to send data across to another program or application. In order to enable this feature, the executable has to be built with ``-DMFIX_MPMD = yes``. - -Inputs ------- - -The following inputs must be preceded by "mfix." and control frequency and the data that will be sent to the other program. - -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| | Description | Type | Default | -+======================+=======================================================================+=============+===========+ -| mpmd_int | Frequency at which data is sent to the other application | Int | -1 | -| | when using MPMD. | | | -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| mpmd_per_approx | Time period for sending data to the other pipeline (approximate) | Real | -1 | -| | when using MPMD. | | | -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| mpmd_static_mfs | A list of the names of static multifabs that will be sent | String | None | -| | only once when using MPMD. Possible values are | | | -| | "volfrac" and "centroid". | | | -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ -| mpmd_mfs | A list of the names of multifabs that will be sent per set frequency | String | None | -| | when using MPMD. Possible values are "ep_g", "vel_g", "T_g" | | | -| | and "X_gk". | | | -+----------------------+-----------------------------------------------------------------------+-------------+-----------+ + +The AMReX-MPMD interface is used to send data to another program or application. In order to enable this feature, the executable +has to be built with ``-DMFIX_MPMD = yes``. + +:ref:`Input parameters controling the AMReX-MPMD interface` are defined in the User Guide run-time inputs section. + Sample Python Program ---------------------- @@ -91,7 +75,7 @@ Receive Once Receive Until *End* ~~~~~~~~~~~~~~~~~~~ -* Receive ``End Flag`` on the python root from the C++ root and broadbast to all python +* Receive ``End Flag`` on the python root from the C++ root and broadcast to all python ranks. If the flag is ``1``, break out of the loop. .. code-block:: python @@ -107,7 +91,7 @@ Receive Until *End* if end == 1: break -* Receive ``Reals`` on the python root from the C++ root and broadbast to all python +* Receive ``Reals`` on the python root from the C++ root and broadcast to all python ranks. Save ``time`` to an array on the python root for plotting. .. code-block:: python @@ -212,11 +196,3 @@ Finalize amr.finalize() amr.MPMD_Finalize() - - - -Limitations ------------- - -* Does not work with simulations that require restarts. -* Cannot be used to send and receive particle data. diff --git a/docs/source_docs/user_guide/inputs/mpmd.rst b/docs/source_docs/user_guide/inputs/mpmd.rst new file mode 100644 index 0000000..8954654 --- /dev/null +++ b/docs/source_docs/user_guide/inputs/mpmd.rst @@ -0,0 +1,37 @@ +.. _InputsMPMD: + +Multiple Program Multiple Data (MPMD) +===================================== + +MFIX-Exa utilizes the AMReX-MPMD interface to send data across to another program or application. In order to enable this +feature, the executable has to be built with ``-DMFIX_MPMD = yes``. + +*Limitations* + +* Does not work with simulations that require restarts. +* Cannot be used to send and receive particle data. + + +The following inputs are defined using the ``mfix`` prefix and control frequency and the data sent to the other program. + ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| | Description | Type | Default | ++======================+=======================================================================+=============+===========+ +| mpmd_int | Frequency at which data is sent to the other application | Int | -1 | +| | when using MPMD. | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_per_approx | Time period for sending data to the other pipeline (approximate) | Real | -1 | +| | when using MPMD. | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_static_mfs | A list of the names of static ``MultiFabs`` that will be sent | String | None | +| | only once when using MPMD. Possible values are | | | +| | "volfrac" and "centroid". | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ +| mpmd_mfs | A list of the names of multifabs that will be sent per set frequency | String | None | +| | when using MPMD. Possible values are "ep_g", "vel_g", "T_g" | | | +| | and "X_gk". | | | ++----------------------+-----------------------------------------------------------------------+-------------+-----------+ + +The description of a python script that gathers and plots velocity statistics for fluid pipe flow is provided in +:ref:`the reference section`, while the python script can be found in the source code under the +tutorials directory, ``tutorials/mpmd/main.py``. diff --git a/docs/source_docs/user_guide/inputs/output.rst b/docs/source_docs/user_guide/inputs/output.rst index 7ef1612..4b9bad4 100644 --- a/docs/source_docs/user_guide/inputs/output.rst +++ b/docs/source_docs/user_guide/inputs/output.rst @@ -11,4 +11,3 @@ Stuff here? output/monitors output/ascent output/spatial_avgs - output/mpmd diff --git a/docs/source_docs/user_guide/run-time_inputs.rst b/docs/source_docs/user_guide/run-time_inputs.rst index 1e3a339..17e22db 100644 --- a/docs/source_docs/user_guide/run-time_inputs.rst +++ b/docs/source_docs/user_guide/run-time_inputs.rst @@ -57,4 +57,5 @@ keywords such as ``mfix``, ``amr``, ``geometry``, ``nodal_proj`` etc. inputs/boundary_conditions inputs/output inputs/advanced + inputs/mpmd -- GitLab From 0a2694617cf37a907fca6fb35c9dc109b1726b30 Mon Sep 17 00:00:00 2001 From: Jordan Musser Date: Tue, 7 Jan 2025 12:16:00 -0500 Subject: [PATCH 4/4] Fix typo in prev. commit --- docs/source_docs/references/mpmd.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source_docs/references/mpmd.rst b/docs/source_docs/references/mpmd.rst index b27ac3d..782847c 100644 --- a/docs/source_docs/references/mpmd.rst +++ b/docs/source_docs/references/mpmd.rst @@ -7,7 +7,7 @@ Multiple Program Multiple Data (MPMD) The AMReX-MPMD interface is used to send data to another program or application. In order to enable this feature, the executable has to be built with ``-DMFIX_MPMD = yes``. -:ref:`Input parameters controling the AMReX-MPMD interface` are defined in the User Guide run-time inputs section. +:ref:`Input parameters controlling the AMReX-MPMD interface` are defined in the User Guide run-time inputs section. Sample Python Program -- GitLab