2. Parallel Framework for Unstructured Meshes (ParFUM)

2.2. Program Structure, Compilation and Execution 

ParFUM is built on Charm++ so you must begin by downloading the latest source version of Charm++ from http://charm.cs.illinois.edu/. Build the source by running ./build and answering the interactive prompts, or by manually specifying the configuration you want to the build script. Make sure to build the Charm++ libraries, not just the core system.

In a charm installation, see charm/examples/ParFUM/ for example and test programs.

2.2.2. Structure of a Typical ParFUM Program 

A typical ParFUM program consists of two functions: init() and driver. The init() function runs only on the first processor, and typically does specialized I/O and startup tasks. In most ParFUM programs init() is primarily used to read in a serial mesh. Once init() completes, ParFUM partitions the mesh and distributes it among all processors. Then driver() is called for every chunk on every processor and performs the main work of the program. This program structure is shown in Figure 40. In the language of the TCHARM manual, init() runs in the serial context and driver() runs in the parallel context.

../_images/parfum_structure.png — 40 A typical ParFUM program consists of an `init()` function running in serial context and a `driver()` function running in parallel context.

In pseudocode, a simple ParFUM program would have the following structure:

     subroutine init
          read the serial mesh and configuration data
     end subroutine
/* after init, the FEM framework partitions the mesh */
     subroutine driver
          get local mesh chunk
          time loop
               FEM computations
               communicate boundary conditions
               more FEM computations
          end time loop
     end subroutine

2.2.3. ParFUM Programs without init/driver 

Although ParFUM provides the init/driver structure as a convenience to the programmer, you can write a ParFUM program without using init or driver. This is a more flexible approach, but it is more complicated than an init/driver program.

In pseudocode, a ParFUM program with a stand-alone main function might look like this:

main program
   MPI_Init
   FEM_Init(MPI_COMM_WORLD)
   if (I am master processor)
      read mesh
   partition mesh
   time loop
       FEM computations
       communicate boundary conditions
       more FEM computations
   end time loop
end main program

In this mode, the FEM framework does not set a default reading or writing mesh, and does no partitioning; you must use the FEM_Mesh routines to create and partition your mesh. See the AMPI manual for details on how to declare the main routine, or the file main.C in ParFUM for an example of how to write a stand-alone main routine. Compiling a ParFUM program without init or driver requires slightly different link flags than a typical ParFUM program, see the compilation section for details.

2.2.4. Compilation 

To compile and link a ParFUM program, you must first have a working copy of Charm++ and the ParFUM libraries. The process for downloading and building this software is described in section 2.2.1.

To compile a FEM program, compile and link using charmc, and pass the flag -language ParFUM to charmc when linking. If your program uses its own main function rather than init and driver, pass -language AMPI instead.

2.2.5. Execution 

At runtime, a Charm++/FEM framework program accepts the following options, in addition to all the usual Charm++ options described in the Charm++ “Installation and Usage Manual”.

+vp \(v\)

Create \(v\) mesh chunks, or “virtual processors”. By default, the number of mesh chunks is equal to the number of physical processors (set with +p \(p\)).
-write

Skip driver(). After running init() normally, the framework partitions the mesh, writes the mesh partitions to files, and exits. As usual, the +vp \(v\) option controls the number of mesh partitions.

This option is only used in programs with an init function.
-read

Skip init(). The framework reads the partitioned input mesh from files and calls driver(). Together with -write, this option allows you to separate out the mesh preparation and partitioning phase from the actual parallel solution run.

This can be useful, for example, if init() requires more memory to hold the unpartitioned mesh than is available on one processor of the parallel machine. To avoid this limitation, you can run the program with -write on a machine with a lot of memory to prepare the input files, then copy the files and run with -read on a machine with a lot of processors.

-read can also be useful during debugging or performance tuning, by skipping the (potentially slow) mesh preparation phase. This option is only used in programs with a driver function.
+tcharm_trace fem

Give a diagnostic printout on every call into the ParFUM framework. This can be useful for locating a sudden crash, or understanding how the program and framework interact. Because printing the diagnostics can slow a program down, use this option with care.

2.3. ParFUM API Reference 

TERRY

2.3.1. Utilities 

ISAAC

2.3.2. Basic Mesh Operations 

TERRY

2.3.3. Mesh Entities 

TERRY

2.3.3.1. Nodes

TERRY

2.3.3.2. Elements

TERRY

2.3.3.3. Sparse Elements

TERRY

2.3.3.4. Mesh Entity Operations

TERRY

2.3.3.5. Mesh Entity Queries

TERRY

2.3.3.6. Advanced Mesh Entity Operations

TERRY

2.3.4. Meshes 

TERRY

2.3.4.1. Basic Mesh Operations

TERRY

2.3.4.2. Mesh Utilities

TERRY

2.3.4.3. Advanced Mesh Operations

TERRY

2.3.5. Mesh Communication: Ghost Layers 

SAYANTAN

2.3.5.1. Ghost Numbering

SAYANTAN

2.3.5.2. Ghost Layer Creation

SAYANTAN

2.3.5.3. Symmetries and Ghosts: Geometric Layer

SAYANTAN

2.3.5.4. Advanced Symmetries and Ghosts: Lower Layer

SAYANTAN

2.3.6. Older Mesh Operations 

SAYANTAN

2.3.6.1. Mesh Data Operations

SAYANTAN

2.3.6.2. Ghost Numbering

SAYANTAN

2.3.6.3. Backward Compatibility

SAYANTAN

2.3.6.4. Sparse Data

SAYANTAN

2.3.7. Mesh Modification 

AARON

2.3.8. Topological Mesh Data 

A ParFUM application can request that the ParFUM framework compute topological adjacencies. All ParFUM applications initially specify the mesh as a set of elements, each element defined by a fixed number of nodes. ParFUM can compute and maintain other sets of adjacencies such as which elements are adjacent to a given node, or which nodes are adjacent(they are both associated with a single element), or which elements share an edge/face with another element. Currently only a single element type is supported, and that element must be FEM_ELEM+0. To generate the structures storing the other types of adjacencies, each process in the ParFUM application should call the following subroutines:

FEM_Add_elem2face_tuples(int mesh, 0, nodesPerFace, numFacesPerElement, faces); specifies the topology of an element, specifically the configuration of its faces(if 3D) or edges(if 2D). Two elements are adjacent if they share a common face. The parameter faces is an integer array of length \(nodesPerFace \cdot numFacesPerElement\). The description is the same as used for determining ghost layers in section 2.3.5.2.

FEM_Mesh_allocate_valid_attr(int mesh, int entity_type);

FEM_Mesh_create_node_elem_adjacency(int mesh);

FEM_Mesh_create_node_node_adjacency(int mesh);

FEM_Mesh_create_elem_elem_adjacency(int mesh);

These subroutines can be called in init on a sequential mesh, or after partitioning in driver. The adjacencies will contain references to ghost elements if the subroutines were called in driver when ghosts are used. The indexes to ghosts are negative integers which can easily be converted to positive indices by using the function FEM_To_ghost_index(id). The C header ParFUM_internals.h is required to be included by the ParFUM application to access the adjacencies. The functions to access the adjacencies are in sections 2.3.8.1, 2.3.8.2, and 2.3.8.3.

The internal data structures representing the adjacencies are maintained correctly when the adaptivity operations described in section 2.3.9 are used.

2.3.8.1. Accessing Element to Element Adjacencies

void e2e_getAll(int e, int *neighbors); places all of element e’s adjacent element ids in neighbors; assumes neighbors is already allocated to correct size

int e2e_getNbr(int e, short idx); returns the id of the idx-th adjacent element

2.3.8.2. Accessing Node to Element Adjacencies

n2e_getLength(int n) returns the number of elements adjacent to the given node n.

n2e_getAll(int n, int *&adjelements, int &sz) for node n place all the ids for adjacent elements into adjelements. You can ignore sz if you correctly determine the length beforehand.

2.3.8.3. Accessing Node to Node Adjacencies

n2n_getLength(int n) returns the number of nodes adjacent to the given node n.

n2n_getAll(int n, int *&adjnodes, int &sz) for node n place all the ids for adjacent nodes into adjnodes. You can ignore sz if you correctly determine the length beforehand.

2.3.9. Mesh Adaptivity 

2.3.9.1. Initialization

If a ParFUM application wants to use parallel mesh adaptivity, the first task is to call the initialization routine from the driver function. This creates the node and element adjacency information that is essential for the adaptivity operations. It also initializes all the mesh adaptivity related internal objects in the framework.

void FEM_ADAPT_Init(int meshID)

Initializes the mesh defined by meshID for the mesh adaptivity operations.

2.3.9.2. Preparing the Mesh

For every element entity in the mesh, there is a desired size entry for each element. This entry is called meshSizing. This meshSizing entry contains a metric that determines element quality. The default metric is the average of the length of the three edges of an element. ParFUM provides various mechanisms to set this field. Some of the adaptive operations use these metrics to maintain quality. In addition, there is another metric which is computed for each element and maintained during mesh adaptivity. This metric is the ratio of the longest side to the shortest altitude, and this value is not allowed to go beyond a certain limit in order to maintain element quality.

void FEM_ADAPT_SetElementSizeField(int meshID, int elem, double size);

For the mesh specified by meshID, for the element elem, we set the desired size for each element to be size.

void FEM_ADAPT_SetElementSizeField(int meshID, double \*sizes);

For the mesh specified by meshID, for the element elem, we set the desired size for each element from the corresponding entry in the sizes array.

void FEM_ADAPT_SetReferenceMesh(int meshID);

For each element int this mesh defined by meshID set its size to the average edge length of the corresponding element.

void FEM_ADAPT_GradateMesh(int meshID, double smoothness);

Resize mesh elements to avoid jumps in element size. That is, avoid discontinuities in the desired sizes for elements of a mesh by smoothing them out. Algorithm based on h-shock correction, described in Mesh Gradation Control, Borouchaki et al.

2.3.9.3. Modifying the Mesh

Once the elements in the mesh have been prepared by specifying their desired sizes, we are ready to use the actual adaptivity operations. Currently we provide Delaunay flip operations, edge bisect operations and edge coarsen operations, all of which are implemented in parallel. We provide several higher level functions which use these basic operations to generate a mesh with higher quality elements while achieving the desired sizing.

void FEM_ADAPT_Refine(int meshID, int qm, int method, double factor,double \*sizes);

Perform refinements on the mesh specified by meshId. Tries to maintain/improve element quality by refining the mesh as specified by a quality measure qm. If method = 0, refine areas with size larger than factor down to factor If method = 1, refine elements down to sizes specified in the sizes array. In this array each entry corresponds to the corresponding element. Negative entries in sizes array indicate no refinement.

void FEM_ADAPT_Coarsen(int meshID, int qm, int method, double factor,double \*sizes);

Perform refinements on the mesh specified by meshId. Tries to maintain/improve element quality by coarsening the mesh as specified by a quality measure qm. If method = 0, coarsen areas with size smaller than factor down to factor If method = 1, coarsen elements up to sizes specified in the sizes array. In this array each entry corresponds to the corresponding element. Negative entries in sizes array indicate no coarsening.

void FEM_ADAPT_AdaptMesh(int meshID, int qm, int method, double factor,double \*sizes);

This function has the same set of arguments as required by the previous two operations, namely refine and coarsen. This function keeps using the above two functions until we have all elements in the mesh with as close to the desired quality. Apart from using the above two operations, it also performs a mesh repair operation which gets rid of some bad quality elements by Delaunay flip or coarsening as the geometry in the area demands.

int FEM_ADAPT_SimpleRefineMesh(int meshID, double targetA, double xmin, double ymin, double xmax, double ymax);

A region is defined by (xmax, xmin, ymax, ymin) and the target area to be achieved for all elements in this region in the mesh specified by meshID is given by targetA. This function only performs a series of refinements on the elements in this region. If the area is larger, then no coarsening is done.

int FEM_ADAPT_SimpleCoarsenMesh(int meshID, double targetA, double xmin, double ymin, double xmax, double ymax);

A region is defined by (xmax, xmin, ymax, ymin) and the target area to be achieved for all elements in this region in the mesh specified by meshID is given by targetA. This function only performs a series of coarsenings on the elements in this region. If the area is smaller, then no refinement is done.