DivideEtImpera

Master: Develop:

Divide

Domain division is achieved through METIS

Impera

Ghost vertices indices may be provided to PETSc in order to exchange data between processes

An interface to Parallel CGNS easily writes results to the output file

---

Dependencies

To build, it is necessary

• g++/gcc 7.4 (at least)
• Open MPI 2.1.1 (at least)
• cmake 3.14.4 (at least)
• make
• HDF5 1.10.0 (at least)
• CGNS 4.1.2
• Boost 1.74.0
• METIS 5.1.0
• MSHtoCGNS 7.0.0

Once you have installed the first three dependecies, you may install Boost, HDF5, CGNS, METIS and MSHtoCGNS by executing setup.sh located in Zeta/Setup/. This script will install shared libraries in release variant.

---

Installation

Simply execute

$ mkdir build
$ cd build
$ cmake .. -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=TRUE
$ make
$ make test
$ make install

---

Domain division

Use CgnsReader to get a GridDataPtr from your input file

boost::mpi::communicator world;
GridDataPtr globalGridData;
if (world.rank() == 0) {
    CgnsReader cgnsReader(std::string inputFile);
    globalGridData = cgnsReader.gridData;
}

Only the root process must pass the GridDataPtr to DomainDivider

SubdomainDataPtr subdomainData;
if (world.rank() == 0) {
    DomainDivider domainDivider;
    subdomainData = domainDivider.divideDomain(GridDataPtr globalGridData);
}
else {
    DomainDivider domainDivider;
    subdomainData = domainDivider.divideDomain();
}

In turn, SubdomainData holds the output of the domain division as well as the information necessary to exchange data between subdomains

struct SubdomainData {
    SubdomainData() = default;
    SubdomainData(GridDataPtr localGridData, std::vector<int> globalIndices);

    GridDataPtr localGridData;
    std::vector<int> globalIndices;

    std::vector<int> ghostGlobalIndices;
    std::pair<int, int> ownershipRange;

    std::vector<std::vector<int>> verticesOfSubdomains;
    std::vector<std::vector<int>> elementsOfSubdomains;
    std::vector<int> subdomainSizes;
};

Where

• localGridData: subdomain GridDataPtr
• globalIndices: global indices of each subdomain vertex, that includes both local and ghost vertices
• ghostGlobalIndices: global indices of the subdomain ghost vertices - it may be passed to VecCreateGhost
• ownershipRange: first and last vertices owned by the subdomain - pass it to ParallelCgnsWriter

---

Parallel output

First, SubdomainCreator is used to create an output file in which each subdomains holds a contiguous partition of vertices

std::string outputFile;
if (world.rank() == 0) {
    DomainCreator domainCreator(GridDataPtr globalGridData, SubdomainDataPtr subdomainData, std::string outputPath);
    outputFile = domainCreator.getFileName();
}
boost::mpi::broadcast(world, outputFile, 0);

Then, ParallelCgnsWriter is put to use

ParallelCgnsWriter parallelCgnsWriter(std::string outputFile, std::string gridLocation, std::pair<int, int> ownershipRange);
parallelCgnsWriter.writeSolution(std::string solutionName);
parallelCgnsWriter.writeField(std::string fieldName, const std::vector<double>& values);

Each subprocess will write the fields values regarding its local vertices

---

Import

DivideEtImpera may be easily imported to any project using CMake, just make sure FindDivideEtImpera.cmake is on your CMAKE_MODULE_PATH. Hence, you may use this snippet

set (DEI_DIR $ENV{DEI_DIR}/${BUILD_TYPE}/${LIBRARY_TYPE})
find_package (DivideEtImpera REQUIRED)
if (DEI_FOUND)
    include_directories (${DEI_INCLUDE_DIR})
endif ()

...

target_link_libraries (${_target} ${DEI_LIBRARIES})

Usually,

$ENV{DEI_DIR} is set to a directory such as /home/felipe/Libraries/dei-0.0.0

${BUILD_TYPE} is the lower case ${CMAKE_BUILD_TYPE} - release OR debug

${LIBRARY_TYPE} is the lower case library type - shared OR static

You may also set ${DEI_DIR} (on your project's CMakeLists.txt) to the installation directory of DivideEtImpera.