- Parallel Framework for Unstructured Meshes (ParFUM)
- Program Structure, Compilation and Execution
- ParFUM API Reference
ParFUM is built on Charm++ so you must begin by downloading the latest
source version of Charm++ from
http://charm.cs.illinois.edu/. Build the
source by running
./build and answering the interactive prompts, or
by manually specifying the configuration you want to the build script.
Make sure to build the Charm++ libraries, not just the core system.
In a charm installation, see charm/examples/ParFUM/ for example and test programs.
A typical ParFUM program consists of two functions:
init() function runs only on the first processor,
and typically does specialized I/O and startup tasks. In most ParFUM
init() is primarily used to read in a serial mesh. Once
init() completes, ParFUM partitions the mesh and distributes it
among all processors. Then
driver() is called for every chunk on
every processor and performs the main work of the program. This program
structure is shown in Figure 55. In the
language of the TCHARM manual,
init() runs in the serial context and
driver() runs in the parallel context.
In pseudocode, a simple ParFUM program would have the following structure:
subroutine init read the serial mesh and configuration data end subroutine /* after init, the FEM framework partitions the mesh */ subroutine driver get local mesh chunk time loop FEM computations communicate boundary conditions more FEM computations end time loop end subroutine
Although ParFUM provides the init/driver structure as a convenience to the programmer, you can write a ParFUM program without using init or driver. This is a more flexible approach, but it is more complicated than an init/driver program.
In pseudocode, a ParFUM program with a stand-alone main function might look like this:
main program MPI_Init FEM_Init(MPI_COMM_WORLD) if (I am master processor) read mesh partition mesh time loop FEM computations communicate boundary conditions more FEM computations end time loop end main program
In this mode, the FEM framework does not set a default reading or writing mesh, and does no partitioning; you must use the FEM_Mesh routines to create and partition your mesh. See the AMPI manual for details on how to declare the main routine, or the file main.C in ParFUM for an example of how to write a stand-alone main routine. Compiling a ParFUM program without init or driver requires slightly different link flags than a typical ParFUM program, see the compilation section for details.
To compile and link a ParFUM program, you must first have a working copy of Charm++ and the ParFUM libraries. The process for downloading and building this software is described in section 5.2.1.
To compile a FEM program, compile and link using
charmc, and pass
-language ParFUM to charmc when linking. If your program
uses its own
main function rather than init and driver, pass
-language AMPI instead.
At runtime, a Charm++/FEM framework program accepts the following options, in addition to all the usual Charm++ options described in the Charm++ “Installation and Usage Manual”.
Create \(v\) mesh chunks, or “virtual processors”. By default, the number of mesh chunks is equal to the number of physical processors (set with
Skip driver(). After running init() normally, the framework partitions the mesh, writes the mesh partitions to files, and exits. As usual, the
+vp\(v\) option controls the number of mesh partitions.
This option is only used in programs with an
Skip init(). The framework reads the partitioned input mesh from files and calls driver(). Together with
-write, this option allows you to separate out the mesh preparation and partitioning phase from the actual parallel solution run.
This can be useful, for example, if init() requires more memory to hold the unpartitioned mesh than is available on one processor of the parallel machine. To avoid this limitation, you can run the program with
-writeon a machine with a lot of memory to prepare the input files, then copy the files and run with
-readon a machine with a lot of processors.
-readcan also be useful during debugging or performance tuning, by skipping the (potentially slow) mesh preparation phase. This option is only used in programs with a
Give a diagnostic printout on every call into the ParFUM framework. This can be useful for locating a sudden crash, or understanding how the program and framework interact. Because printing the diagnostics can slow a program down, use this option with care.
22.214.171.124. Sparse Elements¶
126.96.36.199. Mesh Entity Operations¶
188.8.131.52. Mesh Entity Queries¶
184.108.40.206. Advanced Mesh Entity Operations¶
220.127.116.11. Basic Mesh Operations¶
18.104.22.168. Mesh Utilities¶
22.214.171.124. Advanced Mesh Operations¶
126.96.36.199. Ghost Numbering¶
188.8.131.52. Ghost Layer Creation¶
184.108.40.206. Symmetries and Ghosts: Geometric Layer¶
220.127.116.11. Advanced Symmetries and Ghosts: Lower Layer¶
18.104.22.168. Mesh Data Operations¶
22.214.171.124. Ghost Numbering¶
126.96.36.199. Backward Compatibility¶
188.8.131.52. Sparse Data¶
A ParFUM application can request that the ParFUM framework compute
topological adjacencies. All ParFUM applications initially specify the
mesh as a set of elements, each element defined by a fixed number of
nodes. ParFUM can compute and maintain other sets of adjacencies such as
which elements are adjacent to a given node, or which nodes are
adjacent(they are both associated with a single element), or which
elements share an edge/face with another element. Currently only a
single element type is supported, and that element must be
FEM_ELEM+0. To generate the structures storing the other types of
adjacencies, each process in the ParFUM application should call the
FEM_Add_elem2face_tuples(int mesh, 0, nodesPerFace, numFacesPerElement, faces);
specifies the topology of an element, specifically the configuration of
its faces(if 3D) or edges(if 2D). Two elements are adjacent if they
share a common face. The parameter faces is an integer array of length
\(nodesPerFace \cdot numFacesPerElement\). The description is the
same as used for determining ghost layers in section
FEM_Mesh_allocate_valid_attr(int mesh, int entity_type);
These subroutines can be called in
init on a sequential mesh, or
after partitioning in
driver. The adjacencies will contain
references to ghost elements if the subroutines were called in
driver when ghosts are used. The indexes to ghosts are negative
integers which can easily be converted to positive indices by using the
FEM_To_ghost_index(id). The C header
is required to be included by the ParFUM application to access the
adjacencies. The functions to access the adjacencies are in sections
The internal data structures representing the adjacencies are maintained correctly when the adaptivity operations described in section 5.3.9 are used.
184.108.40.206. Accessing Element to Element Adjacencies¶
void e2e_getAll(int e, int *neighbors); places all of element e’s
adjacent element ids in neighbors; assumes
neighbors is already
allocated to correct size
int e2e_getNbr(int e, short idx); returns the id of the idx-th
220.127.116.11. Accessing Node to Element Adjacencies¶
n2e_getLength(int n) returns the number of elements adjacent to the
n2e_getAll(int n, int *&adjelements, int &sz) for node
all the ids for adjacent elements into
adjelements. You can ignore
sz if you correctly determine the length beforehand.
18.104.22.168. Accessing Node to Node Adjacencies¶
n2n_getLength(int n) returns the number of nodes adjacent to the
n2n_getAll(int n, int *&adjnodes, int &sz) for node
n place all
the ids for adjacent nodes into
adjnodes. You can ignore sz if you
correctly determine the length beforehand.
If a ParFUM application wants to use parallel mesh adaptivity, the first task is to call the initialization routine from the driver function. This creates the node and element adjacency information that is essential for the adaptivity operations. It also initializes all the mesh adaptivity related internal objects in the framework.
void FEM_ADAPT_Init(int meshID)
Initializes the mesh defined by meshID for the mesh adaptivity operations.
22.214.171.124. Preparing the Mesh¶
For every element entity in the mesh, there is a desired size entry for each element. This entry is called meshSizing. This meshSizing entry contains a metric that determines element quality. The default metric is the average of the length of the three edges of an element. ParFUM provides various mechanisms to set this field. Some of the adaptive operations use these metrics to maintain quality. In addition, there is another metric which is computed for each element and maintained during mesh adaptivity. This metric is the ratio of the longest side to the shortest altitude, and this value is not allowed to go beyond a certain limit in order to maintain element quality.
void FEM_ADAPT_SetElementSizeField(int meshID, int elem, double size);
For the mesh specified by meshID, for the element elem, we set the desired size for each element to be size.
void FEM_ADAPT_SetElementSizeField(int meshID, double \*sizes);
For the mesh specified by meshID, for the element elem, we set the desired size for each element from the corresponding entry in the sizes array.
void FEM_ADAPT_SetReferenceMesh(int meshID);
For each element int this mesh defined by meshID set its size to the average edge length of the corresponding element.
void FEM_ADAPT_GradateMesh(int meshID, double smoothness);
Resize mesh elements to avoid jumps in element size. That is, avoid discontinuities in the desired sizes for elements of a mesh by smoothing them out. Algorithm based on h-shock correction, described in Mesh Gradation Control, Borouchaki et al.
126.96.36.199. Modifying the Mesh¶
Once the elements in the mesh have been prepared by specifying their desired sizes, we are ready to use the actual adaptivity operations. Currently we provide Delaunay flip operations, edge bisect operations and edge coarsen operations, all of which are implemented in parallel. We provide several higher level functions which use these basic operations to generate a mesh with higher quality elements while achieving the desired sizing.
void FEM_ADAPT_Refine(int meshID, int qm, int method, double
Perform refinements on the mesh specified by meshId. Tries to maintain/improve element quality by refining the mesh as specified by a quality measure qm. If method = 0, refine areas with size larger than factor down to factor If method = 1, refine elements down to sizes specified in the sizes array. In this array each entry corresponds to the corresponding element. Negative entries in sizes array indicate no refinement.
void FEM_ADAPT_Coarsen(int meshID, int qm, int method, double
Perform refinements on the mesh specified by meshId. Tries to maintain/improve element quality by coarsening the mesh as specified by a quality measure qm. If method = 0, coarsen areas with size smaller than factor down to factor If method = 1, coarsen elements up to sizes specified in the sizes array. In this array each entry corresponds to the corresponding element. Negative entries in sizes array indicate no coarsening.
void FEM_ADAPT_AdaptMesh(int meshID, int qm, int method, double
This function has the same set of arguments as required by the previous two operations, namely refine and coarsen. This function keeps using the above two functions until we have all elements in the mesh with as close to the desired quality. Apart from using the above two operations, it also performs a mesh repair operation which gets rid of some bad quality elements by Delaunay flip or coarsening as the geometry in the area demands.
int FEM_ADAPT_SimpleRefineMesh(int meshID, double targetA, double xmin,
double ymin, double xmax, double ymax);
A region is defined by (xmax, xmin, ymax, ymin) and the target area to be achieved for all elements in this region in the mesh specified by meshID is given by targetA. This function only performs a series of refinements on the elements in this region. If the area is larger, then no coarsening is done.
int FEM_ADAPT_SimpleCoarsenMesh(int meshID, double targetA, double xmin,
double ymin, double xmax, double ymax);
A region is defined by (xmax, xmin, ymax, ymin) and the target area to be achieved for all elements in this region in the mesh specified by meshID is given by targetA. This function only performs a series of coarsenings on the elements in this region. If the area is smaller, then no refinement is done.
We provide a checking function that can be used for debugging purposes to identify corrupted meshes or low quality elements.
void FEM_ADAPT_TestMesh(int meshID);
This provides a series of tests to determine the consistency of the mesh specified by meshID.
188.8.131.52. Index List Calls¶
184.108.40.206. Advanced Index List Calls¶
220.127.116.11. Layout Routines¶
18.104.22.168. Advanced Layout Routines¶
22.214.171.124. Layout Compatibility Routines¶
126.96.36.199. Communication Routines¶
188.8.131.52. Advanced Communication Routines¶