Domain Decomposition

NOTE: This page deals with domain decomposition as practiced in AstroBEAR 1.0. AstroBEAR 2.0 uses a more balanced approach to decomposition based on space-filling curves. As such, this page is considered outdated.

—BRANDON

When a domain is decomposed, it is split into smaller grids. This allows the problem domain to be broken up and parceled out to multiple processors, at the cost of creating additional ghost zones.

There are several schemes of domain decomposition. The simplest is discussed below. The other methods are not yet discussed on this page.

Simple Domain Decomposition in AstroBEAR

Simple domain decomposition in AstroBEAR is controlled through the domain.data file, where a decomposition ratio is set for each dimension of the grid. This ratio tells AstroBEAR how many subdomains to make in the given dimension. For example, a two-dimensional problem domain with a decomposition setting of 4 2 in domain.data will split the domain into eight chunks. Note that these chunks may not be of equal size. If the user requests, for example, 4 subdomains in a direction that has 37 cells, one of the subdomains will be 1 row of cells larger to account for the non-integer decomposition (37/4!=a whole number).

Optimal Decompositions

In theory, there is nothing stopping you from decomposing right up to N_{decompositions}=N_{computational cells}. Remember, however, that decompositions are meant to distribute the problem among multiple processors. In practice, we have found that decomposing the problem into one domain per processor is the optimal ratio in terms of performance; more than that just creates unnecessary ghost zones, and fewer than that results in wasted worker CPU time.

We have also found that changing the dimensions of the decomposition also affects performance. A square or cubic pattern (all dimensions having the same decomposition ratio) is typically a safe bet. If that's not a feasible option, try minimizing the number of decompositions along the direction of the greatest change (for example, if your simulation fires a jet or a shock horizontally, try a 1 x 2 decomposition instead of 2 x 1).

Finally, remember that decomposition creates ghost zones, which can generate a lot of MPI traffic. This can be particularly painful if you're running on a high-latency cluster, where an increase in message passing time can more than cancel out the decrease in computation time. It might be good to play around with a smaller or lower-resolution model to find the optimal decomposition if you aren't sure.

Bisection Decomposition

"Smart" Decomposition