Introducing AstroBEAR 2.0

AstroBEAR In Brief

The growing size of scientific simulations can no longer be accommodated simply by increasing the number of nodes in a cluster. Completing larger jobs without increasing the wall time requires a decrease in the workload per processor (i.e., increased parallelism). Unfortunately, increased parallelism often leads to increased communication time. Minimizing the cost of this communication requires efficient parallel algorithms to manage the distributed AMR structure and calculations.

AstroBEAR's strength lies in its distributed tree structure. Many AMR codes replicate the entire AMR tree on each computational node. This approach incurs a heavy communication cost as the tree continuously broadcasts structural changes to all processors. AstroBEAR, on the other hand, only keeps as much tree information as the local grids need to communicate with processors containing nearby grid regions. This approach saves us memory usage as well as communication time, leaving us well-positioned to take advantage of massively parallel low-memory architectures such as ​BlueGene systems and ​GPUs.

In our distributed tree system, processors can have children and parents in much the same way that grids do. All of a processor's tree information—grids, overlaps, neighbors, and processors—comes from the parent processor. Thus, each processor only needs to communicate with its parent processor and the processors whose data directly interact with it. Grids are distributed among processors via a ​Hilbert ordering, allowing AstroBEAR to take full advantage of the cluster's topology.

Under this new system, a processor has four classes of processors with which it interacts:

• Parent: The processor associated with parent grids of the local processor's grids.
• Child: A processor associated with the children of the local processor's grids.
• Neighbor: A processor associated with one or more grids adjacent to the local processor's grids.
• Overlap: As an AMR simulation evolves, grids are constantly being created and destroyed as the refined regions change. An overlap processor is associated with grids from the previous step that overlap the local processor's current grids.

Workload is managed on AstroBEAR through careful work scheduling and distributed load balance calculations. AstroBEAR calculates a given level's workload for each processor based on the total workload for the level and the processor's remaining workload from the previous level. This approach allows us to balance the workload globally rather than level by level. As a result, AstroBEAR can handle simulations with extremely coarse base grids and many levels of AMR.