wiki:ModulesOnAstroBear

Version 48 (modified by Jonathan, 8 years ago) ( diff )

Writing Modules in AstroBEAR 2.0


AstroBEAR problem modules are stored in the modules directory. Each problem module gets its own directory; when compiling the code, the user creates a symbolic link Problem to the appropriate directory. For more information about setting up a problem directory, see Setting up and Compiling A Problem for more details.

The contents of the problem directory are up to the author, but at least one file must be present: problem.f90. This is the source file that contains the routines referenced by module_control.f90. The tutorial below is basically a discussion of how to fill this module out.

When a problem directory is checked into AstroBEAR, it is usually checked in with a number of .data files. From the standpoint of this tutorial, the most important is problem.data. This is where all of the user-defined variables are stored; these are typically read in by the ProblemModuleInit() routine in problem.f90.

Throughout this page we will be using the following convention: the new module file you are creating is called problem.f90. We will be also be assuming that the user is working in the $(ASTROBEAR)/modules directory.


Creating a New Module In AstroBEAR


This creates a module file and makes sure that AstroBEAR can link to it. The actual writing of the module and defining its interfaces is a separate process described below.

Note that these instructions assume that you are in modules.

  1. Create a new Directory NewModule in modules.
  1. Create a symbolic link to this file with the command ln -s NewModule Problem. Note, that if the Problem symbolic link already exists, you may have to first remove it and then make it anew.
  1. Create a file NewModule/problem.f90. This is the Fortran 90 file where your code will be.


Integrating your Module Into AstroBEAR


Each problem module contains six main subroutines that are required in order for the module to interface with the the AMR engine. These subroutines are listed below.

The basics of writing these modules will be explained below under "Module Basics."

The six subroutines can be split into two groups

Those that modify an individual AMR patch

Note these may get called multiple times on a given processor or not at all depending on how patches are distributed among the processors. The only assignment statements should be for variables local to each subroutine or the fluid data belonging to the info patch. Do not modify module variables or global variables within these subroutines.

  • ProblemGridInit(Info): This is where you initialize the data arrays. Remember that this routine is getting called on a grid-by-grid basis, so attempts to initialize data outside this grid will probably cause a segfault. For an example routine see ProblemGridInit
  • ProblemBeforeStep(Info): The before-step subroutine. This procedure is called before each time-step of the simulation, so that the driving mechanisms specified by the user can be reapplied at each step. For example, a jet simulation would inject new material into the system during ProblemBeforeStep(). Note that no integration occurs during this step; this is just where new conditions are introduced (and sometimes renewed). If you have no special pre-step needs, then leave this routine as a stub.
  • ProblemAfterStep(Info): The after-step subroutine. This procedure is called following each time step of a system. This is perhaps the least commonly used of the control subroutines, but divergence cleaning, special output files, and other post-processing operations could be performed here. If you have no special post-step instructions, then just leave this routine as a stub.
  • ProblemSetErrFlag(Info): Flags regions where this module requires additional resolution. If you do not have any special refinement needs, then just leave this routine as a stub.

And those that get called on by every processor at certain times.

Note here you can adjust module variables or modify the properties of Objects. You can also add global communication here if you desire though this is usually not necessary.

  • ProblemModuleInit(): Module variables are initialized here. This routine is where problem.data are read in and module-level namelists are populated and is called once per processor at the start (or restart) of a simulation. ProblemModuleInit() is also a popular place to put sanity checks that verify the correctness of all inputs. ProblemModuleInit() is also a common place to initialize source terms.
  • ProblemBeforeGlobalStep(level): This is called by every processor one per step per AMR level. If you need to modify variables other than the grid data - this is the correct place to do so. If you are using threading however - you should only modify variables between root steps (when level == 0).

Here is the bare minimum problem module

MODULE Problem
  IMPLICIT NONE
  SAVE
  PUBLIC ProblemModuleInit, ProblemGridInit, ProblemBeforeStep, &
         ProblemAfterStep, ProblemSetErrFlag, ProblemBeforeGlobalStep
  PRIVATE

CONTAINS

  SUBROUTINE ProblemModuleInit()
  END SUBROUTINE ProblemModuleInit

  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemGridInit

  SUBROUTINE ProblemBeforeStep(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemBeforeStep

  SUBROUTINE ProblemAfterStep(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemAfterStep

  SUBROUTINE ProblemSetErrFlag(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemSetErrFlag

  SUBROUTINE ProblemBeforeGlobalStep(n)
     INTEGER :: n
  END SUBROUTINE ProblemBeforeGlobalStep

END MODULE Problem



AstroBEAR Module Basics


Units and Scaling

Astrophysical problems involve many different physical units and constants with a wide range of scales. To avoid overflow or underflow - we scale our units into computational units before storing them in the data arrays. Note with double precision this would be quite rare - but it still convenient to work within physical units appropriate to the problem.

Usually, the physical scales are defined by a few parameters in the physics.data file —you simply enter the scales for density, temperature, velocity, etc in cgs units, and AstroBEAR will read them in. Note that nScale is in cm{-3} and TempScale is in Kelvin. When the code runs it will calculate other scales such as the computational time scale, mass scale, magnetic field scale, and so on which are then dumped to a scales.data file in your run directory. All of those units will be in cgs with the magnetic field scale in gauss. It is often useful to know the computational time scale before running your job as this will affect the final time specified in global.data. The computational time scale can be calculated as follows:

where you may have to first derive

You have two options for making sure that you only put scaled quantities in the data arrays: you can scale your input values before you enter them into your input file (and then assume that you are reading in scaled quantities), or you can use physical quantities in your input files and then scale them within your problem module:

scaled_qty = physical_qty / physical_scale

Either way, a good sanity check is to print out the physical quantities your program uses after the problem is set up. This verifies that the values you think are going in are the values that are actually getting used.


Initializing a Grid / Updating boundary conditions

Initializing a grid involves taking a spatially-constructed problem setup and discretizing it so that it fits nicely in an array. This process is easiest to explain by dissecting an example, such as the one below, where we are trying to initialize the grid with a uniform background (density = 1, pressure = 1) and an overdense spherical clump centered at the origin with radius 1. We then want to add a constant wind of density 1, pressure 1, and velocity 10 coming from the left boundary.

  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef) :: Info
    INTEGER :: i,j,k
    REAL(KIND=xPREC) :: pos(3)

    ! Initialize background
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), 1)=1d0
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivx)=0d0
    IF (ivy /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivy)=0d0
    IF (ivz /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivz)=0d0
    IF (iE /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), iE)=gamma7    

    ! Increase density for points inside of clump
    DO i=1, Info%mX(1)
      DO j=1, Info%mX(2)
        DO k=1, Info%mX(3)
          pos=CellPos(Info, i, j, k)
          IF (sqrt(sum(pos**2)) < 1d0) THEN
            Info%q(i,j,k,irho) = 10d0
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE

  SUBROUTINE ProblemBeforeStep(Info)
    TYPE(InfoDef) :: Info
    INTEGER :: i,j,k, mbc(3)
    REAL(KIND=xPREC) :: pos(3)
    
    !determine number of ghost zones for each dimension
    mbc=levels(Info%level)%gmbc(levels(Info%level)%step)*merge((/1,1,1/),(/0,0,0/),nDim>=(/1,2,3/))

    ! Initialize wind in leftmost boundary
    DO i=1-mbc(1), Info%mX(1)+mbc(1)
      DO j=1-mbc(2), Info%mX(2)+mbc(2)
        DO k=1-mbc(3), Info%mX(3)+mbc(3)
          pos=CellPos(Info, i, j, k)
          IF (pos(1) < GxBounds(1,1)) THEN
            Info%q(i,j,k,irho) = 1d0
            Info%q(i,j,k,ivx)=10d0
            IF (iE /= 0) Info%q(i,j,k, iE)=gamma7+50d0
            IF (ivy /= 0) Info%q(i,j,k, ivy)=0d0
            IF (ivz /= 0) Info%q(i,j,k, ivz)=0d0
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE

Now let's say we want to be able to adjust the density of the clump, the radius of the clump, and the wind velocity at run-time. To do this we need to declare three variables within our module..

MODULE Problem
  USE GlobalDeclarations
  USE PhysicsDeclarations
  USE DataDeclarations
  IMPLICIT NONE
  SAVE
  PUBLIC ProblemModuleInit, ProblemGridInit, ProblemBeforeStep, &
         ProblemAfterStep, ProblemSetErrFlag, ProblemBeforeGlobalStep
  PRIVATE 
  REAL(KIND=qPREC) :: rho, radius, velocity

CONTAINS

  SUBROUTINE ProblemModuleInit()
    NAMELIST/ProblemData/ rho, radius, velocity
    OPEN(UNIT=PROBLEM_DATA_HANDLE, FILE='problem.data', STATUS="OLD")
    READ(PROBLEM_DATA_HANDLE,NML=ProblemData)
    CLOSE(PROBLEM_DATA_HANDLE)
  END SUBROUTINE
  
  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef) :: Info
    INTEGER :: i,j,k
    REAL(KIND=xPREC) :: pos(3)

    ! Initialize background
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), 1)=1d0
    Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivx)=0d0
    IF (ivy /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivy)=0d0
    IF (ivz /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), ivz)=0d0
    IF (iE /= 0) Info%q(1:Info%mX(1),1:Info%mX(2),1:Info%mX(3), iE)=gamma7    

    ! Increase density for points inside of clump
    DO i=1, Info%mX(1)
      DO j=1, Info%mX(2)
        DO k=1, Info%mX(3)
          pos=CellPos(Info, i, j, k)
          IF (sqrt(sum(pos**2)) < radius) THEN
            Info%q(i,j,k,irho) = rho
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE

  SUBROUTINE ProblemBeforeStep(Info)
    TYPE(InfoDef) :: Info
    INTEGER :: i,j,k, mbc(3)
    REAL(KIND=xPREC) :: pos(3)
    
    !determine number of ghost zones for each dimension
    mbc=levels(Info%level)%gmbc(levels(Info%level)%step)*merge((/1,1,1/),(/0,0,0/),nDim>=(/1,2,3/))

    ! Initialize wind in leftmost boundary
    DO i=1-mbc(1), Info%mX(1)+mbc(1)
      DO j=1-mbc(2), Info%mX(2)+mbc(2)
        DO k=1-mbc(3), Info%mX(3)+mbc(3)
          pos=CellPos(Info, i, j, k)
          IF (pos(1) < GxBounds(1,1)) THEN
            Info%q(i,j,k,irho) = 1d0
            Info%q(i,j,k,ivx)=velocity
            IF (iE /= 0) Info%q(i,j,k, iE)=gamma7+half*velocity**2
            IF (ivy /= 0) Info%q(i,j,k, ivy)=0d0
            IF (ivz /= 0) Info%q(i,j,k, ivz)=0d0
          END IF                  
        END DO
      END DO
    END DO
  END SUBROUTINE

  SUBROUTINE ProblemAfterStep(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemAfterStep

  SUBROUTINE ProblemSetErrFlag(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemSetErrFlag

  SUBROUTINE ProblemBeforeGlobalStep(n)
     INTEGER :: n
  END SUBROUTINE ProblemBeforeGlobalStep

END MODULE

There are a lot of other ways we could modify this simple example to have the clump be located anywhere, to have a density profile that is smoothed at the edge, to be a different temperature, or move with a particular velocity, etc… Fortunately, clumps are a commonly used object (as are uniform backgrounds and winds) and there are modules designed to assist users in easily creating clumps, uniform backgrounds, and winds. For example the above module could be rewritten using an Ambient Object, a Clump Object, and a Wind Object.

MODULE Problem
  USE GlobalDeclarations
  USE DataDeclarations
  USE Clumps
  USE Ambients
  USE Winds
  IMPLICIT NONE
  SAVE
  PUBLIC ProblemModuleInit, ProblemGridInit, ProblemBeforeStep, &
         ProblemAfterStep, ProblemSetErrFlag, ProblemBeforeGlobalStep
  PRIVATE 
  REAL(KIND=qPREC) :: rho, radius, velocity

CONTAINS

  SUBROUTINE ProblemModuleInit()
    TYPE(AmbientDef), POINTER :: Ambient
    TYPE(ClumpDef), POINTER :: Clump
    TYPE(WindDef), POINTER :: Wind
    NAMELIST/ProblemData/ rho, radius
    OPEN(UNIT=PROBLEM_DATA_HANDLE, FILE='problem.data', STATUS="OLD")
    READ(PROBLEM_DATA_HANDLE,NML=ProblemData)
    CLOSE(PROBLEM_DATA_HANDLE)


    CALL CreateAmbient(Ambient)

    CALL CreateClump(Clump)
    Clump%density=rho
    Clump%radius=radius
    CALL UpdateClump(Clump)

    CALL CreateWind(Wind)
    Wind%velocity=velocity
    CALL UpdateWind(Wind)

  END SUBROUTINE
  
  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE
 
  SUBROUTINE ProblemBeforeStep(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemBeforeStep

  SUBROUTINE ProblemAfterStep(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemAfterStep

  SUBROUTINE ProblemSetErrFlag(Info)
    TYPE(InfoDef) :: Info
  END SUBROUTINE ProblemSetErrFlag

  SUBROUTINE ProblemBeforeGlobalStep(n)
     INTEGER :: n
  END SUBROUTINE ProblemBeforeGlobalStep

END MODULE

So what is going on here?

And we're done. We don't have to worry about dx or mx or cell positions, or the number of ghost zones etc… All of that detailed work is done for us. It is important to note that the order that objects are created is the same order they are placed on the grid. So had we created the clump object first - the clump data would have been overwritten by the ambient module and there would be no clump.

If we want to get more complicated - we can modify other clump/wind/ambient attributes. All of the available (and default) options for the various objects should be documented on the AstroBearObjects page.

To try the problem module we just built, we can copy the data files from $(ASTROBEAR)/modules/Template directory to the current problem directory $(ASTROBEAR)/modules/Problem and add these three lines to problem.data

&ProblemData
rho = 10.0
radius = 1.0
velocity = 10.0
/

If we follow the procedure Setting up and Compiling A Problem to try to compile and run our new problem module. If you set the dimensions and final time in global.data as

nDim     = 2                ! number of dimensions for this problem (1-3)
GmX      = 30,30,1          ! Base grid resolution [x,y,z]
MaxLevel = 0                        ! Maximum level for this simulation (0 is fixed grid)

and

final_time        = 15d-1               ! The final time in computational units.
final_frame       = 20              ! The final frame [10]

and follow First Run to run it. Following Chapter 2 to use VisIT to analyze the results, we can get a movie like this ClumpMovie


Flagging Cells for Refinement

Some modules may need specific regions refined, regardless of whether or not there is any obvious gradients etc. AstroBEAR flags cells for refinement using the array

  Info%ErrFlag(1:mx,1:my,1:mz)

To clear the cell at (i,j,k), simply set Info%ErrFlag(i,j,k) to 0. An error flag of 0 means that the cell does not need to be refined. This is in general, not a good idea since a previous routine might have already flagged that cell for refinement with good reason. To mark a cell for refinement, set Info%ErrFlag(i,j,k) to 1. The place to do this is in the ProblemSetErrFlag() routine; most conventional physical criteria for refinement are already handled by AstroBEAR itself. For more information see ControllingRefinement for more information as well as ProblemSetErrFlag for an example of how to use this subroutine.


A few notes on magnetic aux fields

Aux fields are particularly difficult to work with - and if initialized improperly (either form a non-divergenceless physical model or from 2nd order errors due to estimating face averages with face-centered values) will produce a probably small but not insignificant divergence that will stick around for the course of the simulation. The easiest way to avoid divergence in your B-fields is to first calculate the vector potential and then to take the curl discretely. In 2D, this means calculating the value of the vector potential at cell corners (which only has a z-component)- and then differencing them in y to get Bx and differencing them in x to get -By. In 3D, the vector potential should be averaged along each edge of the component parallel to that edge. For example Ax should be averaged along each edge that is parallel to the x-axis. The 2nd order errors due to estimating the average value along an edge by the midpoint will not produce divergence in the resulting B-field.

Let's suppose in we want to initialize the grid with a B-field By=sin(x). Well the vector potential would just be Az=cos(x) and our ProblemGridInit routine would look like:

  SUBROUTINE ProblemGridInit(Info)
    TYPE(InfoDef) :: Info
    INTEGER :: i,j,k
    REAL(KIND=qPREC) :: pos(3), dx
    dx = levels(Info%level)%dx
    IF (MaintainAuxArrays) THEN
      Info%aux(1:Info%mX(1)+1,1:Info%mX(2), 1:Info%mX(3), 1)=0d0
      IF (nDim == 3) Info%aux(1:Info%mX(1),1:Info%mX(2), 1:Info%mX(3)+1, 3)=0d0
      DO i=1, Info%mX(1)
        DO j=1, Info%mX(2)+1
          DO k=1, Info%mX(3)
            pos=CellPos(Info, i, j, k)          
            Info%aux(i,j,k,2)=(cos(pos(1)+half*dx)-cos(pos(1)-half*dx))/dx
          END DO
        END DO
      END DO
      DO i=1, Info%mX(1)
        DO j=1, Info%mX(2)
          DO k=1, Info%mX(3)
            Info%q(i,j,k,iBx)=0d0
            Info%q(i,j,k,iBy)=half*(Info%aux(i,j,k,2)+Info%aux(i,j+1,k,2))
            Info%q(i,j,k,iBz)=0d0
            Info%q(i,j,k,iE)=gamma7+half*Info%q(i,j,k,iBy)**2
          END DO
        END DO
      END DO
    ELSE
      DO i=1, Info%mX(1)
        DO j=1, Info%mX(2)
          DO k=1, Info%mX(3)
            pos=CellPos(Info, i, j, k)
            Info%q(i,j,k,iBx)=0d0
            Info%q(i,j,k,iBy)=sin((pos(1))
            Info%q(i,j,k,iBz)=0d0
          END DO
        END DO
      END DO
    END IF
  END SUBROUTINE

Summary

Writing a problem module can be extremely complicated, but for first time users writing relatively simple modules, here are a few helpful tips:


Attachments (3)

Note: See TracWiki for help on using the wiki.