# 3.4 Running applications in parallel

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

This section describes how to run OpenFOAM in parallel on distributed processors. The method of parallel computing used by OpenFOAM is known as domain decomposition, in which the geometry and associated fields are broken into pieces and allocated to separate processors for solution. The process of parallel computation involves: decomposition of mesh and fields; running the application in parallel; and, post-processing the decomposed case as described in the following sections. The parallel running uses the public domain openMPI implementation of the standard message passing interface (MPI).

## 3.4.1 Decomposition of mesh and initial field data

The mesh and fields are decomposed using the decomposePar utility. The underlying aim is to break up the domain with minimal effort but in such a way to guarantee a fairly economic solution. The geometry and fields are broken up according to a set of parameters specified in a dictionary named decomposeParDict that must be located in the system directory of the case of interest. An example decomposeParDict dictionary can be copied from the interFoam/damBreak tutorial if the user requires one; the dictionary entries within it are reproduced below:

17
18  numberOfSubdomains 4;
19
20  method          simple;
21
22  simpleCoeffs
23  {
24      n               ( 2 2 1 );
25      delta           0.001;
26  }
27
28  hierarchicalCoeffs
29  {
30      n               ( 1 1 1 );
31      delta           0.001;
32      order           xyz;
33  }
34
35  manualCoeffs
36  {
37      dataFile        "";
38  }
39
40  distributed     no;
41
42  roots           ( );
43
44
45  // ************************************************************************* //


The user has a choice of four methods of decomposition, specified by the method keyword as described below.

simple
Simple geometric decomposition in which the domain is split into pieces by direction, e.g. 2 pieces in the $x$ direction, 1 in $y$ etc.
hierarchical
Hierarchical geometric decomposition which is the same as simple except the user specifies the order in which the directional split is done, e.g. first in the $y$ direction, then the $x$ direction etc.
scotch
Scotch decomposition which requires no geometric input from the user and attempts to minimise the number of processor boundaries. The user can specify a weighting for the decomposition between processors, through an optional processorWeights keyword which can be useful on machines with differing performance between processors. There is also an optional keyword entry strategy that controls the decomposition strategy through a complex string supplied to Scotch. For more information, see the source code file:

$FOAM_SRC/decompositionMethods/decompositionMethods/scotchDecomp/scotchDecomp.C manual Manual decomposition, where the user directly specifies the allocation of each cell to a particular processor. For each method there are a set of coefficients specified in a sub-dictionary of decompositionDict, named <method>Coeffs as shown in the dictionary listing. The full set of keyword entries in the decomposeParDict dictionary are explained in Table 3.4. Compulsory entries  numberOfSubdomains Total number of subdomains$ N $method Method of decomposition simple/ hierarchical/ scotch/ metis/ manual/ simpleCoeffs entries n Number of subdomains in$ x $,$ y $,$ z $($ n_x  n_y  n_z \$)

delta Cell skew factor Typically, 10−3 hierarchicalCoeffs entries n Number of subdomains in x, y, z (nx ny nz) delta Cell skew factor Typically, −3 10 order Order of decomposition xyz/xzy/yxz… scotchCoeffs entries processorWeights (optional) List of weighting factors for allocation of cells to processors; <wt1> is the weighting factor for processor 1, etc.; weights are normalised so can take any range of values. (<wt1>…<wtN>) strategy Decomposition strategy (optional); defaults to "b" manualCoeffs entries dataFile Name of file containing data of allocation of cells to processors "<fileName>" Distributed data entries (optional) — see section 3.4.3 distributed Is the data distributed across several disks? yes/no roots Root paths to case directories; <rt1> is the root path for node 1, etc. (<rt1>…<rtN>)

Table 3.4: Keywords in decompositionDict dictionary. The decomposePar utility is executed in the normal manner by typing

decomposePar


On completion, a set of subdirectories will have been created, one for each processor, in the case directory. The directories are named processorN where N = 0,1, ... represents a processor number and contains a time directory, containing the decomposed field descriptions, and a constant/polyMesh directory containing the decomposed mesh description.

## 3.4.2 Running a decomposed case

A decomposed OpenFOAM case is run in parallel using the openMPI implementation of MPI. openMPI can be run on a local multiprocessor machine very simply but when running on machines across a network, a file must be created that contains the host names of the machines. The file can be given any name and located at any path. In the following description we shall refer to such a file by the generic name, including full path, <machines>.

The <machines> file contains the names of the machines listed one machine per line. The names must correspond to a fully resolved hostname in the \etc\hosts file of the machine on which the openMPI is run. The list must contain the name of the machine running the openMPI. Where a machine node contains more than one processor, the node name may be followed by the entry cpu=n where n is the number of processors openMPI should run on that node.