......@@ -9,7 +9,7 @@
Grid Creation
-------------
To run MFiX-Exa you must specify :cpp:`n_cell` in the inputs file --
To run MFIX-Exa you must specify :cpp:`n_cell` in the inputs file --
this is the number of cells spanning the domain in each coordinate direction at level 0.
Users often specify :cpp:`max_grid_size` as well. The default load balancing algorithm then divides the
......
......
......@@ -28,3 +28,7 @@ Options supported by AMReX include:
- Round-robin: sort grids and assign them to ranks in round-robin fashion -- specifically
FAB ``i`` is owned by CPU ``i % N`` where N is the total number of MPI ranks.
These methods work for both fluid and particle grids if dual-grid is enabled. MFIX-Exa also supports
a Greedy load balancing algorithm for particle grids. It balances the particle counts per rank and
aligns particle grids with fluid grids to minimize the data-transfer between two grids.
docs/source_docs/images/particle_banner.png

487 KiB

.. MFIX-EXA documentation master file, created Thu Aug 2 12:19:39 2018.
Welcome to MFIX-EXA's documentation!
====================================
MFIX-Exa is a new multiphase computational fluid dynamics (CFD) modeling tool
that can take advantage of massively parallel high performance computers (HPC)
with the goal of scaling well at exascale. It combines the underlying physics
from the existing `MFIX <https://mfix.netl.doe.gov/>`_ code with the
`AMReX <https://amrex-codes.github.io/amrex/>`_ framework, creating the next
generation, high performance, multiphase CFD workhorse.
.. image:: /images/particle_banner.png
MFIX-Exa is under active development as part of the MFIX-Exa
application project in DOE's
`Exascale Computing Project (ECP) <https://www.exascaleproject.org/>`_. All of
MFIX-Exa's development is done in the
`NETL gitlab repository <https://mfix.netl.doe.gov/gitlab/exa/mfix>`_, with
active development in the develop branch. Changes are merged into
the master branch at the beginning of each month.
When mature, this project will provide a modern multi-level multi-grid fluid
solver with chemistry, heat transfer, complex geometry, and two particle models:
1. **Discrete particle method (DEM)** - Each particle is modeled as a sphere,
resolving all particle collisions
2. **Particle in cell method (PIC)** - Particles are group in parcels and see
other parcels through a stress gradient.
To get started with MFIX-Exa, follow the user guide sections:
.. toctree::
:maxdepth: 1
:caption: User guide:
Introduction
GettingStarted_Chapter
Inputs_Chapter
ManagingGridHierarchy_Chapter
To learn more about the implementation, follow the following reference sections:
.. toctree::
:maxdepth: 1
:caption: Reference:
Fluids_Chapter
Particles_Chapter
EB_Chapter
Debugging
The following section detail the testing and benchmarks used to ensure solutions
and performance:
.. toctree::
:maxdepth: 1
:caption: Tests and Benchmarks:
RunningTestSuite
RegressionTesting
CITests
NightlyTests
qb/index
Notice
------
Neither the United States Government nor any agency thereof, nor any of
their employees, makes any warranty, expressed or implied, or assumes
any legal liability or responsibility for the accuracy, completeness, or
usefulness of any information, apparatus, product, or process disclosed
or represents that its use would not infringe privately owned rights.
- MFIX-Exa is provided without any user support for applications in the
user's immediate organization. It should not be redistributed in
whole or in part.
- The use of MFIX-Exa is to be acknowledged in any published paper based on
computations using this software by citing this MFIX-Exa site.
- The authors would appreciate receiving any reports of bugs or other
difficulties with the software, enhancements to the software, and
accounts of practical applications of this software.
.. _Chap:InputsCheckpoint:
Checkpoint/Restart
==================
The following inputs must be preceded by "amr" and control checkpoint/restart.
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+=========================+=======================================================================+=============+===========+
| restart | If present, then the name of file to restart from | String | None |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| check_int | Frequency of checkpoint output; | Int | -1 |
| | if -1 then no checkpoints will be written | | |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| check_file | Prefix to use for checkpoint output | String | chk |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| check_walltime | Write a check point file after the specified walltime (HH:MM:SS) | String | None |
| | has lapsed. For example, if amr.check_waltime = 00:10:00, then a | | |
| | checkpoint file is after a job has run for ten minutes. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| geom_chk_write | When True, writes the EB geometry data into geom_chk_file | bool | False |
| | and additionally, geom_refined_chk_file, if levelset | | |
| | refinement is enabled. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| geom_chk_read | When True, reads the EB geometry data from geom_chk_file | bool | False |
| | and additionally, geom_refined_chk_file, if levelset | | |
| | refinement is enabled. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| geom_chk_file | Name of the EB checkpoint file that is used to store or read | String | geom_chk |
| | the unrefined geometry. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
| geom_refined_chk_file | Name of the EB checkpoint file that is used to store or read | String | geom\_ |
| | the refined geometry, i.e. when levelset refinement is enabled. | | refined\_ |
| | | | chk |
+-------------------------+-----------------------------------------------------------------------+-------------+-----------+
Drag Types
==========
Interphase Coupling
===================
Drag
----
The following inputs must be preceded by "mfix."
......@@ -32,6 +35,7 @@ With the variables defined as follows:
.. code:: shell
/*
* \brief Returns: the calculated drag coefficient.
*
* Inputs:
......@@ -39,7 +43,7 @@ With the variables defined as follows:
* Mug - gas laminar viscosity
* ROpg - gas density * EP_g
* vrel - magnitude of gas-solids relative velocity
* DPM - particle diamater of solids phase M
* DPM - particle diameter of solids phase M
* DPA - average particle diameter
* PHIS - solids volume fraction of solids phases
* fvelx - x component of the fluid velocity at the particle position
......@@ -126,3 +130,37 @@ The Gidaspow model is defined as
{
return 0.0;
}
Heat Transfer Coefficient
-------------------------
The following inputs must be preceded by "mfix."
+-------------------+---------------------------------+-------------+--------------+
| | Description | Type | Default |
+===================+=================================+=============+==============+
| convection_type | Which HTC model to use | String | RanzMarshall |
+-------------------+---------------------------------+-------------+--------------+
The options currently supported in mfix are :cpp:`RanzMarshall` (default) and :cpp:`Gunn`.
In both models the HTC is determined from a Nusslet number corelation.
The RanzMarshall Nusselt number correlation is defined as:
.. code:: shell
amrex::Real N_Nu = 2.0 + 0.6 * std::sqrt(N_Re) * std::pow(N_Pr, 0.333);
The Gunn Nusselt number correlation is defined as:
.. code:: shell
amrex::Real N_Nu =
(7 - 10*EPg + 5*EPg*EPg)*(1 + .7*std::pow(N_Re, 0.2)*std::cbrt(N_Pr))
+ (1.33 - 2.4*EPg + 1.2*EPg*EPg)*std::pow(N_Re, 0.7)*std::cbrt(N_Pr);
.. sec:InputsDiscretization:
Discretization
==============
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
| Key | Description | Type | Default |
+===========================+=======================================================================+=============+==============+
| advection_type | Predictor-Corrector Method of Lines ("mol") or Godunov ("godunov") | String | Godunov |
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
| redistribution_type | Use flux ("FluxRedist"), state ("StateRedist") or no ("NoRedist") | | |
| | redistribution | String | StateRedist |
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
| redistribute_nodal_proj | Redistribute the velocity field after the nodal projection | Bool | False |
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
| use_drag_coeff_in_proj_gp | Algebraically consistent p coeff in proj or (default) simplified form | Bool | False |
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
| use_drag_in_godunov | Include a drag term in the Godunov flux or (default) not | Bool | False |
+---------------------------+-----------------------------------------------------------------------+-------------+--------------+
Notes: The code was originally developed with MOL and FluxRedist. Preliminary
tests show that the new single-step Godunov method is roughly twice as fast as
the predictor-corrector MOL at the same time step (e.g., CFL limited to 0.5).
Further, the Godunov method allows for roughly twice the time step, CFL should
be limited to 0.9 for stability. Finally, it is recommended that the Godunov
method be used in conjunction with StateRedist. While not fully vetted, early
tests also show increased stability in complex geometries for a StateRedist-
Godunov scheme compared to the previous FluxRedist-MOL scheme.
.. _Chap:InputsHypre:
Hypre Inputs
=============
The following inputs control the hypre settings and are read directly
by AMReX when we use hypre as the bottom solver, i.e., when
:cpp:`nodal_proj.bottom_solver = hypre` and/or when
:cpp:`mac_proj.bottom_solver = hypre` (see :ref:`Chap:InputsMultigrid`)
These settings must be preceded by the hypre_namespace setting corresponding
to the solver (see :ref:`Chap:InputsMultigrid`).
NOTE: By default, the MAC and nodal projections use the same settings
specified with the namespace :cpp:`hypre`, however the solvers can be configured
separately by specifying :cpp:`mac_proj.hypre_namespace` and :cpp:`nodal_proj.hypre_namespace`.
When not using the default namespace :cpp:`hypre`, additional restrictions apply:
#. Both :cpp:`mac_proj.hypre_namespace` and :cpp:`nodal_proj.hypre_namespace` must be set.
#. :cpp:`mac_proj.hypre_namespace` and :cpp:`nodal_proj.hypre_namespace` must be unique.
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| | Description | Type | Default |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| verbose | Verbosity of hypre | Int | 0 |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| adjust_singular_matrix | Should be true if the problem to be solved has singular matrix | Bool | false |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_solver | Type of hypre solver | String | BoomerAMG |
| | | | |
| | Options are BoomerAMG, GMRES, COGMRES, LGMRES, FlexGMRES, BiCGSTAB, | | |
| | PCG or Hybrid | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_preconditioner | Type of preconditioner | String | none |
| | | | |
| | Options are BoomerAMG or euclid | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| recompute_preconditioner | Recompute preconditioner during runs | Bool | true |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| write_matrix_files | Write out matrix into text files | Bool | false |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| overwrite_existing_matrix_files | Over-write existing matrix files | Bool | false |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_verbose | Verbosity of BoomerAMG preconditioner | Int | 0 |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_logging | When using BoomerAMG preconditioner | Int | 0 |
| | | | |
| | See HYPRE_BoomerAMGSetLogging | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_coarsen_type | When using BoomerAMG preconditioner | Int | 6 |
| | | | |
| | See HYPRE_BoomerAMGSetCoarsenType | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_cycle_type | When using BoomerAMG preconditioner | Int | 1 |
| | | | |
| | See HYPRE_BoomerAMGSetCycleType | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_relax_type | When using BoomerAMG preconditioner | Int | 6 |
| | | | |
| | See HYPRE_BoomerAMGSetRelaxType | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_relax_order | When using BoomerAMG preconditioner | Int | 1 |
| | | | |
| | See HYPRE_BoomerAMGSetRelaxOrder | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_num_sweeps | When using BoomerAMG preconditioner | Int | 2 |
| | | | |
| | See HYPRE_BoomerAMGSetNumSweeps | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_max_levels | When using BoomerAMG preconditioner | Int | 20 |
| | | | |
| | See HYPRE_BoomerAMGSetMaxLevels | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_strong_threshold | When using BoomerAMG preconditioner | Real | 0.57 |
| | | | |
| | See HYPRE_BoomerAMGSetStrongThreshold | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bamg_interp_type | When using BoomerAMG preconditioner | Int | 0 |
| | | | |
| | HYPRE_BoomerAMGSetInterpType | | |
+-----------------------------------+-----------------------------------------------------------------------+-------------+--------------+
File moved
.. role:: cpp(code)
:language: c++
.. _Chap:InputsLoadBalancing:
Gridding and Load Balancing
......@@ -35,21 +38,24 @@ The following inputs must be preceded by "fabarray_mfiter" and determine how we
The following inputs must be preceded by "particles"
+-------------------+-----------------------------------------------------------------------+-------------+--------------+
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| | Description | Type | Default |
+===================+=======================================================================+=============+==============+
+======================+=======================================================================+=============+==============+
| max_grid_size_x | Maximum number of cells at level 0 in each grid in x-direction | Int | 32 |
| | for grids in the ParticleBoxArray if dual_grid is true | | |
+-------------------+-----------------------------------------------------------------------+-------------+--------------+
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| max_grid_size_y | Maximum number of cells at level 0 in each grid in y-direction | Int | 32 |
| | for grids in the ParticleBoxArray if dual_grid is true | | |
+-------------------+-----------------------------------------------------------------------+-------------+--------------+
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| max_grid_size_z | Maximum number of cells at level 0 in each grid in z-direction | Int | 32 |
| | for grids in the ParticleBoxArray if dual_grid is true. | | |
+-------------------+-----------------------------------------------------------------------+-------------+--------------+
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| tile_size | Maximum number of cells in each direction for (logical) tiles | IntVect | 1024000,8,8 |
| | in the ParticleBoxArray if dual_grid is true. | | |
+-------------------+-----------------------------------------------------------------------+-------------+--------------+
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| reduceGhostParticles | Remove unused ghost particles from communication, | Bool | True |
| | only supported by GPU build | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
Note that when running a granular simulation, i.e., no fluid phase, :cpp:`mfix.dual_grid` must be 0. Hence,
the :cpp:`particles.max_grid_size` (in each direction) have no meaning. Therefore the fluid grid and tile
......@@ -63,14 +69,37 @@ The following inputs must be preceded by "mfix" and determine how we load balanc
+======================+=======================================================================+=============+==============+
| dual_grid | If true then use the "dual_grid" approach for load balancing | Bool | False |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| load_balance_fluid | Only relevant if (dual_grid); if so do we also regrid mesh data | Int | 1 |
| load_balance_fluid | Only relevant if (dual_grid); if so do we also regrid mesh data | Int | 0 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| load_balance_type | What strategy to use for load balancing | String | KnapSack |
| | Options are "KnapSack"or "SFC" | | |
| | Options are "KnapSack", "SFC", or "Greedy" | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| knapsack_weight_type | What weighting function to use if using Knapsack load balancing | String | RunTimeCosts |
| | Options are "RunTimeCosts" or "NumParticles"" | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| knapsack_nmax | Maximum number of grids per MPI process if using knapsack algorithm | Int | 128 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| greedy_dir | The direction in which the greedy algorithm cuts overloaded boxes | Int | 0 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| greedy_min_grid_size | The minimum particle grid size in the greedy load balance algorithm | Int | 2 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| greedy_3d | Partition particle grids in 3D with the greedy algorithhm | Bool | False |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| overload_tolerance* | The ratio between the maximum workload and the average workload | Real | 1.2 |
| | in the greedy algorithm | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| underload_tolerance* | The ratio between the minimum workload and the average workload | Real | 0.8 |
| | in the greedy algorithm | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| grid_pruning | Remove all covered grids from the base mesh; this may result in | Bool | False |
| | disjoined grids | | |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
\* The greedy partitioning algorithm uses the tolerances to set the expected
workload range for each rank but doesn't strictly enforce them. The algorithm
creates a partition as balance as possible even if the partition doesn't
meet the tolerances.
To allow a user to verify the breakdown of fluid grids created before running a full simulation, an input option,
:cpp:`mfix.only_print_grid_report` is supported. By default, it is :cpp:`False`. When set to :cpp:`True`, the run uses
minimal memory to print the grid coverage report and exits immediately after that.
This diff is collapsed.
......@@ -4,10 +4,10 @@ Multigrid Inputs
================
The following inputs can be set directly in the AMReX solver classes but we
set them via the MFiX-Exa routines because we may want different inputs for the
different solvers called by MFiX-Exa.
set them via the MFIX-Exa routines because we may want different inputs for the
different solvers called by MFIX-Exa.
NOTE: the nodal solver settings are read in directly by AMReX,
the MAC and diffusion settings by MFiX.
the MAC and diffusion settings by MFIX-Exa.
These control the nodal projection and must be preceded by "nodal_proj":
......@@ -18,9 +18,9 @@ These control the nodal projection and must be preceded by "nodal_proj":
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_verbose | Verbosity of BiCGStab solver in nodal projection | Int | 0 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| rtol | Relative tolerance in nodal projection | Real | 1.e-11 |
| mg_rtol | Relative tolerance in nodal projection | Real | 1.e-11 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| atol | Absolute tolerance in nodal projection | Real | 1.e-14 |
| mg_atol | Absolute tolerance in nodal projection | Real | 1.e-14 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| maxiter | Maximum number of iterations in the nodal projection | Int | 100 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
......@@ -31,8 +31,16 @@ These control the nodal projection and must be preceded by "nodal_proj":
| | If set to 0, the bottom solver will be called at the current level | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver | Which bottom solver to use in the nodal projection | String | bicgcg |
| | | | |
| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_namespace | Namespace to use in the nodal projection when using hypre | String | hypre |
| | to control hypre specific settings. It can be any string. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_interface | Which interface to use in the nodal projection when using hypre | String | ij |
| | | | |
| | Options are ij, semi_structured or structured | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
These control the MAC projection and must be preceded by "mac_proj":
......@@ -43,9 +51,9 @@ These control the MAC projection and must be preceded by "mac_proj":
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_verbose | Verbosity of BiCGStab solver in MAC projection | Int | 0 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| rtol | Relative tolerance in MAC projection | Real | 1.e-11 |
| mg_rtol | Relative tolerance in MAC projection | Real | 1.e-11 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| atol | Absolute tolerance in MAC projection | Real | 1.e-14 |
| mg_atol | Absolute tolerance in MAC projection | Real | 1.e-14 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| maxiter | Maximum number of iterations in the MAC projection | Int | 200 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
......@@ -56,8 +64,19 @@ These control the MAC projection and must be preceded by "mac_proj":
| | If set to 0, the bottom solver will be called at the current level | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver | Which bottom solver to use in the MAC projection | String | bicgcg |
| | | | |
| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_namespace | Namespace to use in the MAC projection when using hypre | String | hypre |
| | to control hypre specific settings. It can be any string. | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| hypre_interface | Which interface to use in the MAC projection when using hypre | String | ij |
| | | | |
| | Options are ij, semi_structured or structured | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
NOTE: If the :cpp:`nodal_proj` and :cpp:`mac_proj` :cpp:`hypre_namespace`'s are set, they must be distinct unless set to
:cpp:`hypre`, in which case the default behavior is recovered in which case hypre settings apply to all solvers.
These control the diffusion solver and must be preceded by "diffusion":
......@@ -68,9 +87,9 @@ These control the diffusion solver and must be preceded by "diffusion":
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_verbose | Verbosity of BiCGStab solver in diffusion solve | Int | 0 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| rtol | Relative tolerance in diffusion solve | Real | 1.e-11 |
| mg_rtol | Relative tolerance in diffusion solve | Real | 1.e-11 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| atol | Absolute tolerance in diffusion solve | Real | 1.e-14 |
| mg_atol | Absolute tolerance in diffusion solve | Real | 1.e-14 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| maxiter | Maximum number of iterations in diffusion solve | Int | 100 |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
......@@ -81,5 +100,6 @@ These control the diffusion solver and must be preceded by "diffusion":
| | If set to 0, the bottom solver will be called at the current level | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
| bottom_solver | Which bottom solver to use in the diffusion solve | String | bicgcg |
| | Options are bicgcg, bicgstab, cg, cgbicg, smoother or hypre | | |
| | | | |
| | Options are bicgcg, bicgstab, cg, cgbicg, or smoother | | |
+-------------------------+-----------------------------------------------------------------------+-------------+--------------+
......@@ -7,32 +7,34 @@ The following inputs must be preceded by "amr" and control frequency and naming
as whether the EB geometry or level set should be written out, and if the particles should be written out in Ascii
format (for debugging).
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+=====================+=======================================================================+=============+===========+
+======================+=======================================================================+=============+===========+
| plot_int | Frequency of plotfile output; | Int | -1 |
| | if -1 then no plotfiles will be written at this frequency | | |
| plot_per_exact | Time period of plotfile output (exact); will modify dt | Real | -1 |
| | if -1 then no plotfiles will be written at this frequency | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| plot_per_approx | Time period of plotfile output (approximate); does not modify dt | Real | -1 |
| | if -1 then no plotfiles will be written at this frequency | | |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| plotfile_on_restart | Should we write a plotfile when we restart (only used if plot_int>0) | Bool | False |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| plot_file | Prefix to use for plotfile output | String | plt |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| write_ls | Should we write a plotfile holding the level set and volfrac? | Bool | False |
| | If true, it will only be written once,after initialization or restart | | |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| write_eb_surface | Should we write out the EB geometry in vtp format | Bool | False |
| | If true, it will only be written once,after initialization or restart | | |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| par_ascii_file | Prefix to use for ascii particle output | String | par |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| par_ascii_int | Frequency of ascii particle output; | Int | -1 |
| | if -1 then no plotfiles will be written | | |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| | if -1 then no particle ascii files will be written | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| par_ascii_per_approx | Time period of the ascii particle output (approximate); | Real | -1 |
| | if -1 then particle ascii files will not be written at this frequency | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| ascent_int | Frequency of ascent pipeline; | Int | -1 |
| | if -1 then ascent will not be called. | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| ascent_per_approx | Time period of the ascent pipeline (approximate); | Real | -1 |
| | if -1 then ascent will not be called. | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
The following inputs must be preceded by "amr" and control what variables will be written in plotfiles.
......@@ -81,4 +83,101 @@ The following inputs must be preceded by "amr" and control what variables will b
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| plt_phase | Save particle type to plot file | Int | 0 |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
The following inputs must be preceded by "mfix" and control whether the EB geometry or level set should be written out.
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+======================+=======================================================================+=============+===========+
| write_ls | Should we write a plotfile holding the level set and volfrac? | Bool | False |
| | If true, it will only be written once,after initialization or restart | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| write_eb_surface | Should we write out the EB geometry in vtp format | Bool | False |
| | If true, it will only be written once,after initialization or restart | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
| plt_geom | Should we write a plotfile holding the EB geometry data? | Bool | False |
| | If true, it will only be written once,after initialization or restart | | |
+----------------------+-----------------------------------------------------------------------+-------------+-----------+
`Ascent <ascent.readthedocs.io>`_ has been integrated into MFIX-Exa for *in situ* visualization.
For codes that have been built with Ascent support, the following inputs must be preceded by "ascent"
and specifies the ascent actions for fluid and/or particles. The frequency which these are called
is controlled by `ascent_int` or `ascent_per_approx`, see above. Note that if an ascent pipeline
is being included in a GPU build/run, then you must enable managed memory,
i.e., set `amrex.the_arena_is_managed = true`.
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+=====================+=======================================================================+=============+===========+
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
| actions | yaml file of the ascent actions (ex. ascent_actions.yaml). If no file | String | |
| | name is provided, then calls to Ascent are skipped. | | |
+---------------------+-----------------------------------------------------------------------+-------------+-----------+
The following inputs must be preceded by "amr.solids" and allow to write additional plotfiles which contain only solids variables in specific regions at fixed timesteps or approximated simulation times. All these parameters are user-defined.
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| | Description | Type | Default |
+===========================+=================================================================+=============+===========+
| regions | Specify regions that will be used for plotfiles | String | |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region] | Specify which solids phases will be plotted | String | |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plot_int | Specify timestep frequency for plotting the file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plot_per_approx | Specify time interval frequency for ploting the file | Real | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_regtest | Save all variables to plot file (overrides all other IO flags) | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_radius | Save particle radius to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_volume | Save particle volume to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_mass | Save particle mass to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_ro_p | Save particle density to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_omoi | Save inverse of particle momentum of inertia to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_vel_p | Save particle velocity to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_omega_p | Save particle angular velocity to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_statwt | Save particle statistical weight to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_drag_p | Save particle drag force to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_cp_s | Save particle specific heat coefficient to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_T_p | Save particle temperature to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_convection | Save particle convection coefficient to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_X_s | Save particle species mass fractions to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_vel_s_txfr | Save particle interphase velocity transfer to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_h_s_txfr | Save particle interphase enthalpy transfer to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_mass_sn_txfr | Save particle interphase mass transfer to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_phase | Save particle phase to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
| [region].plt_state | Save particle state to plot file | Int | 0 |
+---------------------------+-----------------------------------------------------------------+-------------+-----------+
For each region with name [region] specified in the inputs, it will be saved a plotfile with name "partsXXXXX_[region]", where XXXXX stands for the timestep number.
Below is an example for specifying the inputs to plot solids data in a given
region.
.. code-block:: none
amr.solids.regions = my_region
amr.solids.my_region.plot_int = 10
amr.solids.my_region.plt_vel_p = 1
This diff is collapsed.
......@@ -3,18 +3,20 @@
Time Stepping
=============
The following inputs must be preceded by "mfix." Note that if both are specified, both criteria
Note that if both are specified, both criteria
are used and the simulation still stop when the first criterion is hit. In the case of unsteady flow,
the simulation will stop when either the number of steps reaches max_step or time reaches stop_time.
In the case of unsteady flow, the simulation will stop when either the tolerance (difference between
subsequent steps) is reached or the number of iterations reaches the maximum number specified.
The following inputs must be preceded by ``mfix``.
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| | Description | Type | Default |
| Key | Description | Type | Default |
+======================+=======================================================================+=============+==============+
| max_step | Maximum number of time steps to take | Int | -1 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| stop_time | Maximum time to reach | Real | -1.0 |
| stop_time | Maximum time to reach (s) | Real | -1.0 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| fixed_dt | Should we use a fixed timestep? | Int | 0 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
......@@ -31,7 +33,7 @@ The following inputs must be preceded by "mfix" and are only relevant if running
Currently, the criterion for setting "steady_state" to true is if "dt" is undefined in mfix.dat
+-----------------------+-----------------------------------------------------------------------+-------------+------------+
| | Description | Type | Default |
| Key | Description | Type | Default |
+=======================+=======================================================================+=============+============+
| steady_state | Are we running a steady-state calculation? | Int | 0 |
+-----------------------+-----------------------------------------------------------------------+-------------+------------+
......@@ -46,7 +48,7 @@ Setting the Time Step
---------------------
There are several ways that the inputs are used to determine what time step
is used in the evolution of the fluid-particle system in MFiX-Exa.
is used in the evolution of the fluid-particle system in MFIX-Exa.
1) In a pure particle case, the :cpp:`mfix.fixed_dt`, if specified, is only used to determine the frequency
of outputs, it has no effect on the "dtsolid" used in the particle evaluation. If you do not specify a positive
......
......
......@@ -8,7 +8,7 @@ The following inputs must be preceded by "mfix."
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| | Description | Type | Default |
+======================+=======================================================================+=============+==============+
| verbose | Verbosity in MFiX-Exa routines | Int | 0 |
| verbose | Verbosity in MFIX-Exa routines | Int | 0 |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
| ooo_debug | If true then print the name of the routine we are in | Bool | False |
+----------------------+-----------------------------------------------------------------------+-------------+--------------+
......
......
......@@ -7,7 +7,7 @@
Particle Basics
===============
In MFiX-Exa, particles are managed by the `MFIXParticleContainer
In MFIX-Exa, particles are managed by the `MFIXParticleContainer
<https://amrex-codes.github.io/MFIX-Exa/doxygen/class_m_f_i_x_particle_container.html>`_
class. This class is derived from AMReX's :cpp:`NeighborParticleContainer`
and handles all of the particle data.
......
......
File moved
File moved
.. role:: cpp(code)
:language: c++
Particles on GPUs
==========================
......@@ -50,11 +53,22 @@ The final on-grid neighbor list data structure consists of two arrays. First, we
Note that, because of our use of managed memory to store the particle data and the neighbor list, the above code will work when compiled for either CPU or GPU.
The above algorithm deals with constructing a neighbor list for the particles on a single grid. When domain decomposition is used, one must also make copies of particles on adjacent grids, potentially performing the necessary MPI communication for grids associated with other processes. The routines `fillNeighbors`, which computes which particles needed to be ghosted to which grid, and `updateNeighbors`, which copies up-to-date data for particles that have already been ghosted, have also been offloaded to the GPU, using techniques similar to AMReX's `Redistribute` routine. The important thing for users is that calling these functions does not trigger copying data off the GPU.
The above algorithm deals with constructing a neighbor list for the particles on a single grid.
When domain decomposition is used, one must also make copies of particles on adjacent grids,
potentially performing the necessary MPI communication for grids associated with other processes.
The routines `fillNeighbors`, which computes which particles needed to be ghosted to which grid, and `updateNeighbors`,
which copies up-to-date data for particles that have already been ghosted, have also been offloaded to the GPU,
using techniques similar to AMReX's `Redistribute` routine.
Note that when the particles are dense, only a small portion of the ghost particles are neighbors to the particles
inside the grids. AMReX provides a function `selectActualNeighbors` to filter out the ghost particles that will
not be used for building the neighbor list. So the subsequent calls to `updateNeighbors` can avoid transferring
these unused ghost particles, which significantly reduces the communication cost in some tests.
To use this optimization, set `particles.reduceGhostParticles` to :cpp:`true` in MFIX-Exa's inputs.
Once the neighbor list has been constructed, collisions with both particles and walls can easily be processed.
MFiX-Exa currently runs on both CPU-only and hybrid CPU/GPU architectures.
MFIX-Exa currently runs on both CPU-only and hybrid CPU/GPU architectures.