Writing Modules in AstroBEAR 2.0

AstroBEAR problem modules are stored in the modules directory. Each problem module gets its own directory; when compiling the code, the user creates a symbolic link Problem to the appropriate directory. For more information about setting up a problem directory, see Setting up and Compiling A Problem for more details.

The contents of the problem directory are up to the author, but at least one file must be present: problem.f90. This is the source file that contains the routines referenced by module_control.f90. The tutorial below is basically a discussion of how to fill this module out.

When a problem directory is checked into AstroBEAR, it is usually checked in with a number of .data files. From the standpoint of this tutorial, the most important is problem.data. This is where all of the user-defined variables are stored; these are typically read in by the ProblemModuleInit() routine in problem.f90.

Throughout this page we will be using the following convention: the new module file you are creating is called problem.f90. We will be also be assuming that the user is working in the $(ASTROBEAR)/modules directory.

Creating a New Module In AstroBEAR

This creates a module file and makes sure that AstroBEAR can link to it. The actual writing of the module and defining its interfaces is a separate process described below.

Note that these instructions assume that you are in modules.

1. Create a new Directory NewModule in modules.

2. Create a symbolic link to this file with the command ln -s NewModule Problem. Note, that if the Problem symbolic link already exists, you may have to first remove it and then make it anew.

3. Create a file NewModule/problem.f90. This is the Fortran 90 file where your code will be.

Integrating your Module Into AstroBEAR

Each problem module contains six main subroutines that are required in order for the module to interface with the the AMR engine. These subroutines are listed below.

The basics of writing these modules will be explained below under "Module Basics."

The six subroutines can be split into two groups

Those that modify an individual AMR patch

Note these may get called multiple times on a given processor or not at all depending on how patches are distributed among the processors. The only assignment statements should be for variables local to each subroutine or the fluid data belonging to the info patch. Do not modify module variables or global variables within these subroutines.

• ProblemGridInit(Info): This is where you initialize the data arrays. Remember that this routine is getting called on a grid-by-grid basis, so attempts to initialize data outside this grid will probably cause a segfault. For an example routine see ProblemGridInit
• ProblemBeforeStep(Info): The before-step subroutine. This procedure is called before each time-step of the simulation, so that the driving mechanisms specified by the user can be reapplied at each step. For example, a jet simulation would inject new material into the system during ProblemBeforeStep(). Note that no integration occurs during this step; this is just where new conditions are introduced (and sometimes renewed). If you have no special pre-step needs, then leave this routine as a stub.
• ProblemAfterStep(Info): The after-step subroutine. This procedure is called following each time step of a system. This is perhaps the least commonly used of the control subroutines, but divergence cleaning, special output files, and other post-processing operations could be performed here. If you have no special post-step instructions, then just leave this routine as a stub.
• ProblemSetErrFlag(Info): Flags regions where this module requires additional resolution. If you do not have any special refinement needs, then just leave this routine as a stub.

And those that get called on by every processor at certain times.

Note here you can adjust module variables or modify the properties of Objects. You can also add global communication here if you desire though this is usually not necessary.

• ProblemModuleInit(): Module variables are initialized here. This routine is where problem.data are read in and module-level namelists are populated and is called once per processor at the start (or restart) of a simulation. ProblemModuleInit() is also a popular place to put sanity checks that verify the correctness of all inputs. ProblemModuleInit() is also a common place to initialize source terms.
• ProblemBeforeGlobalStep(level): This is called by every processor one per step per AMR level. If you need to modify variables other than the grid data - this is the correct place to do so. If you are using threading however - you should only modify variables between root steps (when level == 0).

AstroBEAR Module Basics

Simulation Data

All AstroBEAR modules have at least one thing in common: initializing the problem domain. Within our code, the problem domain's data is held in InfoDef structures, which is why so many module subroutines take an InfoDef structure as a parameter. To make use of the InfoDef structure, the following statement is required at the beginning of problem.f90:

USE DataDeclarations

Anything that is defined in the InfoDef type is now available. For example, q need not be defined…just reference it by Info%q.

There are two major data arrays in InfoDef: the q array and the aux array. q holds the volume averaged data and is used by all AstroBEAR simulations while aux holds face averaged data and is used only for MHD when nDim > 1. Note that volume averaged data and cell-centered data are often used interchangeably, but there is an important distinction. Take for instance a simple function f(x) defined on the interval [0:h]. The average of f(x) over the interval is

f(0)+d₁f(0)h/2+d₂f(0)h²/3+…

where the cell centered value is

f(0)+d₁f(0)h/2+d₂f(0)h²/6+…

so the cell centered value is second order accurate for the volume average and usually is a quick way to estimate the volume average. However if the function has large 2nd derivatives (or higher) this can lead to large errors in the volume average. This is often apparent when modeling discontinuities along curved boundaries. There are two ways to handle this problem:

• Introduce smoothing to the physical model to remove large 2nd and higher derivatives
• Better approximate the volume average either by
  • Analytical integration (often non-trivial if possible)
  • Numerical integration (ie sub-sampling)

The q array takes the form q(x,y,z,variable) where variable is an index that refers to the various physical quantities such as density, momentum, energy, etc. in each cell. The order of the quantities in the variable array is dependent on the equation of state, whether or not magnetic fields are being tracked, etc… For 2D hydro (non MHD) the order of the fields is (rho, px, py, E). So if we wanted to set the energy of the cell at integer location i,j,k we would use

Info%q(i,j,k,4) = 1.0

However if we were to change the number of dimensions from 2 to 3, then the order of the fields would be rho, px, py, pz, E and the above statement would not set the energy, but the z momentum to 1.0 and leave the energy unchanged. The solution is to avoid using integer constants for the 4th array index and instead use integer variables that are adjusted based on the equations of state, number of dimensions, etc… These variables are declared in PhysicsDeclarations so we need to also add

USE PhysicsDeclarations

to the top of our module. Then we can set the energy of cell i,j,k regardless of the specifics of our problem by using

Info%q(i,j,k,iE) = 1.0

Also, if we happen to be using an isothermal equation of state, then the energy is no longer stored within the q array and the value of iE is set to 0 to indicate this. So it is generally a good idea to check the value of iE as follows

IF (iE /= 0) Info%q(i,j,k,iE)=1.0

Additional variables used to store slots are:

• irho - density (always non-zero)
• ivx - x momentum (always non-zero)
• ivy - y momentum (non-zero unless 2D, 3D, MHD)
• ivz - z momentum (non-zero unless 3D, MHD)
• iE - Energy (non-zero unless isothermal EOS)
• iBx - x magnetic field (non-zero unless MHD)
• iBy - y magnetic field (non-zero unless MHD)
• iBz - z magnetic field (non-zero unless MHD)

There are also two arrays that are sometimes useful as well

• iB(1:3) = (/iBx, iBy, iBz/)
• imom(1:3) = (/ivx, ivy, ivz/)

The aux array holds face-centered data, and is only used in MHD problems. If you are running a strictly hydrodynamic problem or a hydrodynamic + elliptic problem, then you will not need aux.

Dimensions

The number of cells in the x, y, & z direction for the core region of each Info structure is stored in the array

Info%mX(1:3)

and often one will declare local variables mx, my, & mz to avoid repeatedly having to type Info%mx(d).

mx=Info%mX(1)
my=Info%mX(2)
mz=Info%mX(3)

The data within this core region (which does not include ghost zones) is stored in Info%q(1:mx,1:my,1:mz,1:NrHydroVars) where NrHydroVars represents the number of fluid variables including tracers. If running with fewer than 3 dimensions, the unused dimensions have an extent of 1.

Before we can initialize a cell we must calculate it's physical location and extent. To do so we need to know the cell size for the Info's AMR level. The properties of each level are stored in the levels(:) array. To access this data we must use the GlobalDeclarations module by adding the following to our module at the top.

USE GlobalDeclarations

Then to access properties of level n - for example the current time that level has advanced to we would use levels(n)%tnow. If we wanted to now the current time step for level n we could use levels(n)%dt. And to access the cell size for level n we could use levels(n)%dx. Since the level a given info structure resides on is stored in Info%level, the cell size is given by levels(Info%level)%dx. So to get the x-position of the center of a cell with x-index i we could use

xlower=Info%xBounds(1,1)
dx=levels(Info%level)%dx
x=xlower+(REAL(i)-.5)*dx

Note we subtract 0.5 from the index before multiplying by the spacing since we are calculating the cell center. And that the cell actually goes from x-.5*dx to x+.5*dx. Also note that we convert the integer to a real before subtracting .5. And if we want to calculate x,y,z we could use

xlower=Info%xBounds(1,1)
dx=levels(Info%level)%dx
x=xlower + (REAL(i)-.5)*dx
y=ylower + (REAL(j)-.5) * dx
z=zlower + (REAL(k)-.5) * dx
IF (nDim < 2) y=ylower
IF (nDim < 3) z=zlower

The last two lines are necessary since we don't want to add .5 to the y or z dimensions if we are only in 1D or 2D. We could also streamline this using the Fortran MERGE function and storing (/x,y,z/) in an array pos(:) using

pos=Info%xBounds(:,1)+merge((REAL((/i,j,k/))-.5)*dx, (/0d0,0d0,0d0/), nDim < (/1,2,3/))

Finally since the precision of the various info fields related to spatial position is a parameter xPrec (could be single or double), some compilers will complain unless you convert (/i,j,k/ as well as .5 to the right kind of REAL.

pos=Info%xBounds(:,1)+merge((REAL((/i,j,k/),KIND=xPREC)-half)*dx, (/0d0,0d0,0d0/), nDim < (/1,2,3/))

Note that the variable half is a parameter equal to REAL(.5, KIND=xPREC) declared in GlobalDeclarations

Finally there is a function already called CellPos that does the same calculation which makes life much easier.

 pos=CellPos(Info, i, j, k)

The Info%aux array is a little different. The aux array holds magnetic flux values, which are face-averaged. This means that every volume averaged value in Info%q is bracketed in each dimension by two Info%aux values. To accommodate the extra values, Info%aux is a 1:mx+1 by 1:my+1 by 1:mz+1 box, but the aux dimensions are actually different for each variable:

Bx = Info%aux(1:mx+1, 1:my, 1:mz, 1)
By = Info%aux(1:mx, 1:my+1, 1:mz, 2)
Bz = Info%aux(1:mx, 1:my, 1:mz+1, 3)

The additional cells (the ones in the "upper-front right" corner of the aux array) are not used. To locate the center of the face for the Bx fields, we would subtract half*dx from the cell center.

x_pos=CellPos(Info, i, j, k)-(/half,0d0,0d0/)

and for By and Bz we could use

y_pos=CellPos(Info, i, j, k)-(/0d0,half,0d0/)
z_pos=CellPos(Info, i, j, k)-(/0d0,0d0,half/)

Units and Scaling

Astrophysical problems involve many different physical units and constants with a wide range of scales. To avoid overflow or underflow - we scale our units into computational units before storing them in the data arrays. Note with double precision this would be quite rare - but it still convenient to work within physical units appropriate to the problem.

Usually, the physical scales are defined in the physics.data file —you simply enter the scales for density, temperature, velocity, etc in cgs units, and AstroBEAR will read them in. Note that nScale is in cm^{-3} and TempScale is in Kelvin

You have two options for making sure that you only put scaled quantities in the data arrays: you can scale your input values before you enter them into your input file (and then assume that you are reading in scaled quantities), or you can use physical quantities in your input files and then scale them within your problem module:

scaled_qty = physical_qty / physical_scale

Either way, a good sanity check is to print out the physical quantities your program uses after the problem is set up. This verifies that the values you think are going in are the values that are actually getting used.

Initializing a Grid

Initializing a grid involves taking a spatially-constructed problem setup and discretizing it so that it fits nicely in an array. This process is easiest to explain by dissecting an example, such as the one below, where we are trying to initialize the grid with a uniform background and an overdense spherical clump centered at the origin with radius 1.

Note that during the ProblemGridInit routine, ghost zones do not need to be initialized (rmbc = 0) - however - during beforestep calculations they should be (if needed).

  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef), POINTER :: Info
    INTEGER :: i,j,k
    REAL(KIND=xPREC) :: pos(3)

    ! Initialize background
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), 1)=1d0
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivx)=0d0
    IF (ivy /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivy)=0d0
    IF (ivz /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivz)=0d0
    IF (iE /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), iE)=gamma7    

    ! Increase density for points inside of clump
    DO i=1, Info%mX(1)
      DO j=1, Info%mX(2)
        DO k=1, Info%mX(3)
          pos=CellPos(Info, i, j, k)
          IF (sqrt(sum(pos**2)) < 1d0) THEN
            Info%q(i,j,k,irho) = 10d0
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE

Now let's say we want to be able to adjust the density of the clump and the radius of the clump at run-time. To do this we need to declare two variables within our module..

MODULE Problem
  USE GlobalDeclarations
  USE PhysicsDeclarations
  USE DataDeclarations
  IMPLICIT NONE
  SAVE
  PUBLIC ProblemModuleInit, ProblemGridInit, &
       ProblemBeforeStep, ProblemAfterStep, ProblemSetErrFlag, ProblemBeforeGlobalStep
  PRIVATE REAL(KIND=qPREC) :: rho, radius

CONTAINS

  SUBROUTINE ProblemModuleInit()
    NAMELIST/ProblemData/ rho, radius
    OPEN(UNIT=PROBLEM_DATA_HANDLE, FILE='problem.data', STATUS="OLD")
    READ(PROBLEM_DATA_HANDLE,NML=ProblemData)
    CLOSE(PROBLEM_DATA_HANDLE)
  END SUBROUTINE
  
  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef), POINTER :: Info
    INTEGER :: i,j,k
    REAL(KIND=xPREC) :: pos(3)

    ! Initialize background
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), 1)=1d0
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivx)=0d0
    IF (ivy /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivy)=0d0
    IF (ivz /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivz)=0d0
    IF (iE /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), iE)=gamma7    

    ! Increase density for points inside of clump
    DO i=1, Info%mX(1)
      DO j=1, Info%mX(2)
        DO k=1, Info%mX(3)
          pos=CellPos(Info, i, j, k)
          IF (sqrt(sum(pos**2)) < radius) THEN
            Info%q(i,j,k,irho) = rho
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE
 ...

There are a lot of other ways we could modify this simple example to have the clump be located anywhere, to have a density profile that is smoothed at the edge, to be a different temperature, or move with a particular velocity, etc… Fortunately, clumps are a commonly used astrophysical object and there is a clump object module to assist users in easily creating clumps. For example the above module could be rewritten as:

MODULE Problem
  USE GlobalDeclarations
  USE DataDeclarations
  USE Clumps
  USE Ambients
  IMPLICIT NONE
  SAVE
  PUBLIC ProblemModuleInit, ProblemGridInit, &
       ProblemBeforeStep, ProblemAfterStep, ProblemSetErrFlag, ProblemBeforeGlobalStep
  PRIVATE REAL(KIND=qPREC) :: rho, radius

CONTAINS

  SUBROUTINE ProblemModuleInit()
    TYPE(AmbientDef), POINTER :: Ambient
    TYPE(ClumpDef), POINTER :: Clump
    NAMELIST/ProblemData/ rho, radius
    OPEN(UNIT=PROBLEM_DATA_HANDLE, FILE='problem.data', STATUS="OLD")
    READ(PROBLEM_DATA_HANDLE,NML=ProblemData)
    CLOSE(PROBLEM_DATA_HANDLE)
    CALL CreateAmbient(Ambient)
    CALL CreateClump(Clump)
    Clump%density=rho
    Clump%radius=radius
    CALL UpdateClump(Clump)
  END SUBROUTINE
  
  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef), POINTER :: Info
  END SUBROUTINE
 
 ...

So what is going on here? First we've added USE statements for the Clumps and Ambients modules. Then in our ProblemModuleInit() routine we've declared an ambientdef and a clumpdef pointer. Then in CreateAmbient() we create an ambient object with the default properties - density = 1, pressure = 1, velocity = 0, etc… which sets the entire grid to be uniform. Then we create a clump object and modify its properties (density, and radius) before updated the clump object. And we're done.

And if we want to get more complicated - we can modify other clump attributes.

Each grid (or InfoDef structure) comes with several arrays that describe its geometry:

• mX(3): The grid's overall size. Each element mx(n) represents the number of cells along the nth dimension.
• xbounds(3,2): The spatial coordinates of the grid's lower and upper bounds in each dimension. These values are always given in computational units.
• dX: The size of a spatial step. dX can also be thought of as the size of a cell. In AstroBEAR's current form, dX is the same in every dimension and is therefore not actually an array.

To make this example more readable (and therefore easier to debug), we assigned the various array values more Cartesian-sounding names.

Similarly, when calculating the spatial equivalents of an index, we went with x, y and z. The half-step in the position calculations represents the fact that the spatial position is at the center of the cell, not the left side.

The check on the number of dimensions makes sure that 2D problem does not accidentally get initialized with ghost cells (see below) or a spatial step. This allows us to use the same initialization code for 2D or 3D problems.

The levels() array is a global array structures that contain information specific to each level. You can see in this example that we reference the appropriate level by using the Info%level attribute.

You may have noticed the nested loops don't go from 1 to mX(n). This is because the data array size is not the same as the grid size. The size of each grid along dimension n is mX(n). The data arrays, however, have a number of "ghost cells" associated with them. The number of ghost cells on the end of a grid is given by the quantity

levels(Info%level)%CoarsenRatio * levels(Info%level)%gmbc

These ghost cells only appear along the dimensions of the problem, though, so the data arrays for a 2D problem will not have ghost cells along the third dimension. So the real extents of the data arrays in a 3D problem are:

Info%q(1 - rmbc : mX(1) + rmbc, &
       1 - rmbc : mX(2) + rmbc, &
       1 - rmbc : mX(3) + rmbc, &
       NrVars)

Info%aux(1 - rmbc : mX(1) + rmbc + 1, &
         1 - rmbc : mX(2) + rmbc + 1, &
         1 - rmbc : mX(3) + rmbc + 1, &
         NrVars)

Flagging Cells for Refinement

Some modules may need specific regions refined, regardless of whether or not there is any obvious error there. AstroBEAR flags cells for refinement using the array

Info%ErrFlag(1 - rmbc : mX(1) + rmbc, &
                1 - rmbc : mX(2) + rmbc, &
                1 - rmbc : mX(2) + rmbc, &
                NrVars)

To clear the cell at (i,j,k), simply set Info%ErrFlag(i,j,k) to 0. An error flag of 0 means that the cell does not need to be refined. This is in general, not a good idea since a previous routine might have already flagged that cell for refinement with good reason. To mark a cell for refinement, set Info%ErrFlag(i,j,k) to 1. The best place to do this is in the ProblemSetErrFlag() routine; most conventional physical criteria for refinement are already handled by AstroBEAR itself.

See ProblemSetErrFlag for more details and an example of how to use this subroutine.

Sample Module

The best way to understand how a module works is by example. The following example is a module for the Rayleigh-Taylor instability.

Sample Problem Module​

Summary

Writing a problem module can be extremely complicated, but for first time users writing relatively simple modules, here are a few helpful tips:

• Understand what the five main subroutines are for, especially ProblemModuleInit and ProblemGridInit
• Know which variables come with DataDeclarations, GlobalDeclarations, and PhysicsDeclarations and which variables must be defined in problem.f90
• Understand the form of the q array, and also the aux array if using MHD
• Make sure data is scaled so that only computational units are stored in the data arrays
• Since it is often useful to write a module that can run in 2D or 3D, pay close attention to how nDim affects variables

Domain Objects

While they do not implement actual physics, AstroBEAR domain objects are useful enough to mention here. AstroBEAR 2.0 comes with a Shape object that can be used to define regions of space within a domain. These shape objects can be used by other objects and processes to impose special conditions within regions of the domain, even if they span multiple grids or processors. For more information, visit the AstroBEAR objects tutorial.