Posts for the month of December 2011

Meeting Update

  • Looked at how Athena handles CTU w/ source terms

https://clover.pas.rochester.edu/trac/astrobear/wiki/CodeExplanation/SourceTerms

  • Looked at embedding clumps in flows where both clump and background flow are in pressure equilibrium. (Also updated Clump objects to now support velocities and can appear from off the grid.)

https://clover.pas.rochester.edu/trac/astrobear/attachment/wiki/CollidingFlows/clumpflow10b.gif

  • Talked with Rich about mounting cloverdata, grassdata, and alfalfadata, across machines to hopefully avoid folks running sims and having data be dumped to the department disk. Also discovered we have another machine besides alfalfa (botwin) that was intended to be used as a backup? Dave is still waiting to hear back from LSI about raid controller software

Meeting Update 12.21

See Magnetized Clumps project page for updates.

Running 32/64 processor jobs on the standard queue...

It seems that running 32/64 processor jobs on the standard queue is no longer practically feasible… I believe this is due to the backfill policy favoring the plethora of single processor jobs. The only 32+ job I've gotten to run lately on the standard queue was after the CRC downtime when backfill would not have been a factor. Steve recommended using the ib queue for larger jobs although this queue restricts us to using 14 nodes plus the 8 afrank nodes instead of the 94 nodes available to the standard queue… I have however, gotten a job to run on the ib queue there with a reasonable wait time although the wall time is limited to 2 days instead of 5 so this means more frequent restarts and more babysitting…

Also talked with Eric and we decided to do 3D runs of non-linear initial density perturbations and virialized velocity perturbations

Pew pew pew

Meeting Update 12/13/2011 - Eddie

Work on porting the non-equilibrium cooling routines to astrobear2.0 is under way. I've made some of the obvious changes that were required and am now working through the gritty details.

In regards to the test suite, I've been looking into the OrbitingParticles problem again because the images don't look right, and it's not running correctly any more. When I ran it on alfalfa doing a separate compile and run it worked fine. However, it did not run correctly with the buildproblem command. Hopefully, I'll have this resolved before Friday's demonstration, but we still have a handful of other tests that are working just fine if I don't.

I also just started converting my own hydro code to 2D, but I doubt this will be done any time soon. This is just a side project that probably won't receive much attention for a while.

A little confused about the conversion of gravitational energy to kinetic energy

The last run shows the collapse of a slightly perturbed sphere. I would expect that the total energy is conserved or that KE+U+iE = C… The run is however, isothermal - so any heating due to compression is lost… but does this explain the essentially flat curve that is 2KE+V? https://clover.pas.rochester.edu/trac/astrobear/wiki/GravoTurbulence#VII

So I reran the simulation with a gamma of 1.001 to track this energy. Sure enough the rate of increase in the internal energy is very similar to the kinetic energy during the initial stages of the simulation.

CRL 618 update

The four runs (clump with a stratified ambient, clump with a constant ambient, jet with a stratified ambient and jet with a constant ambient) are all in bluegene's queue. We're also slowly pushing them forward in bluehive, but 5 amr levels require a lot of memory and computation. Here's the lates snapshot f Bin's clump-constantMedium:

http://www.pas.rochester.edu/~martinhe/2011/crl-13dec11.png

yet this run, as the others did, stop do to a potential memory leak. The debugg team is aware of this and they're working on it. See

https://clover.pas.rochester.edu/trac/astrobear/blog/blin12062011

for results of the other runs.

Meeting Update 12.12

New runs on Magnetized Clump Project page. Scroll down to bottom to find five new animations.

PN paper to be submitted to MNRAS

I've started working on this paper again. Seems to me that we have to modify:

-the abstract

-the introduction

-the conclusions

Stable Keplerian disks TABLE

Cylindrical disks (no grav support height), gamma=1.001

Time increases downwards and the images (click to enlarge) are logarithmic edge-on views of density. Note that to have pressure equilibrium I've set tempdisk=tempamb/100, whereas in the binary simulation these temperatures are the same. Also, in order to show as as much info about the structure of disks as possible, the color scale limits are not the same for all maps.

TABLE 1

rsoft=rd/2; rth=2rd rsoft=rd/2; rth=8rd rsoft=rd/2; rth=16rd rsoft=rd/4; rth=8rd rsoft=rd/8; rth=2rd rsoft=rd/8; rth=8rd rsoft=rd/16; rth=16rd
same but 2amr →same but 2 amr →same but 2 amr →same but 3 amr →same →http://www.pas.rochester.edu/~martinhe/2011/disk/ini-4amr.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-7.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-17.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-20.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-28.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft2-therm2-100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft2-therm8-96.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft2-therm16-100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-100.png

Flared disks (support height), gamma=1.001

TABLE 2

rsoft=rd/2; rth=2rd rsoft=rd/2; rth=8rd rsoft=rd/2; rth=16rd rsoft=rd/4; rth=8rd rsoft=rd/8; rth=2rd rsoft=rd/8; rth=8rd rsoft=rd/16; rth=16rd
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f0.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f0.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/ini-4amrf.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/ini-5amrf.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft16-therm16-1.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft16-therm16-f5.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f20.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f20.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f20.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f40.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f40.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f40.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f60.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f60.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f60.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f80.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f80.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f80.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft4-therm8-f100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm2-f100.pnghttp://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f99.png

Flared disks (support height), gamma=1.001, tempd=tempamb (instead of pressd=pressamb):

TABLE 3

rsoft=rd/2; rth=2rd rsoft=rd/2; rth=8rd rsoft=rd/2; rth=16rd rsoft=rd/4; rth=8rd rsoft=rd/8; rth=2rd rsoft=rd/8; rth=8rd rsoft=rd/16; rth=16rd
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f0-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f20-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f40-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f60-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f80-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f100-iso.png

Same as table 3 (rsoft=rd/8; rth=8rd) but for a Plummer gravity soft profile, instead of a spline one:

TABLE 4

http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f0-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f20-isoP.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f40-isoP.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f60-isoP.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f80-isoP.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f100-isoP.png

Here's a movie (clock to enlarge) of the run in table 4 but with the same color bar than the corresponding column in table 3.

This is zoomed out. Note that BC are periodic:

http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f-iso.gif


Same as table 3 (rsoft=rd/8; rth=8rd) but for gamma=1.667, instead of 1.001:

TABLE 5

http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f0-iso.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f20-isoG.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f40-isoG.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f60-isoG.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f80-isoG.png
http://www.pas.rochester.edu/~martinhe/2011/disk/soft8-therm8-f100-isoG.png

Submitted Scrambler Paper!

Hooray :)

Stable Keplerian disks

We want to simulate stable Keplerian disks, here I'll keep record of the tests I'l be doing in this context.

Jonathan: So as I see it, fundamentally there are 8 different parameters that fully define the problem - at least for a fixed grid run

softening length disk height disk radius thermal radius cell size density contrast box size/boundary conditions equation of state (gamma)

I think it makes sense to fix chi >> 1 (like 100) and gamma=5/3 and to set the box length ≥ 4 disk radii to avoid boundary effects and then use periodic bc's (or reflecting)

That leaves only five parameters: softening length disk height disk radius thermal radius cell size

I think we want to keep the softening length small but not too small … Because of numerical diffusion - gravitational energy that is converted into rotational energy inside of a few cells will get converted into heat resulting in jets etc… Keeping the softening length at 4 cells will reduce this effect.

That leaves 4 free parameters disk height disk_radius thermal_radius softening length

Since there is no cooling the problem can be arbitrarily scaled so the disk radius can be fixed without loss of generality and will make setting up the data files easier.

This just leaves 3 more parameters or ratios

disk height / disk radius thermal radius / disk radius softening length / disk radius

With the disk setup - there is no pressure support in the z-direction so it might make sense to have a disk that is not a hockey puck but a rotated wedge where at any given radius, the disk mass can be balanced by thermal support.. GM/r*(h/r) ~ cs2 or h = cs2 * r2 / GM

This would essentially give a disk where the height is a quadratic and would be comparable to the radius at r=GM/cs2 (or at the thermal radius)

This would essentially remove the disk height as a free parameter and would limit it to physically consistent values… We then just have

thermal radius / disk radius softening length / disk radius

Having the thermal radius > disk radius will prevent us from having super puffy disks and having the softening length << disk radius will allow for physically consistent disk regions…

I would suggest doing a set of runs where the thermal radius = 2, 4, 8 disk radii and the softening length = 1/16, 1/8, ¼, and ½ of the disk radius

New rotation movie-

New movie posted of the rotating collapse from a side-on perspective -

http://www.pas.rochester.edu/~erica/sinks.html

Also — modification of the fflags in astrobear 2 happens in Makefile.Inc — use -g and - traceback flags for debugging.

Meeting Update

  • Still working with Dave on getting raid scripts up and running. Clover is optimal - but not sure about grass or alfalfa
  • Submitted paper to astro-ph
  • Worked on setting up clumplets
  • First 3D results from colliding flows
  • Perhaps with postprocessing scripts and various image production capabilities it would be possible to have automated output from simulations go to a web directory with a html template that displays pertinent information from the run
    • computational scales
    • resolution
    • plots of density
    • histograms of various thingies
    • a curve showing the time development of mass, energy, momentum, etc…

We could even have pull down menus for showing various plots quantities - and Kris's handy image scroller ? I think this would save a lot of time in the long run and would provide a convenient way to view data. Of course it would be no substitute for a nice annotated visit plot and having pages on the wiki that discuss the various runs would be good - but folks wouldn't have to generate a lot of content for each run… but it would still all be accessible.

Update 12/06

Martin and I are doing identical runs of CRL618 of stratified clump/jet and not stratified clump/jet. We had problems in Bluehive with using 5 amrs. A ticket was submitted. We are now running these using only 4 amrs at the moment. The sims have only been running for 1 or 2 days, and the movies show what we have so far.

CLUMP - NOT STRATIFIED http://www.pas.rochester.edu/~blin/crl618/1206/clump-noSTRAT.gif

CLUMP - STRATIFIED http://www.pas.rochester.edu/~blin/crl618/1206/clump-STRAT.gif

JET - STRATIFIED http://www.pas.rochester.edu/~blin/crl618/1206/jet-STRAT.gif

JET - NOT STRATIFIED has not starting running in Bluehive yet.

Binary

The binary problem is producing a good looking disk after 1 orbit when using rgrav-soft=0.5:

http://www.pas.rochester.edu/~martinhe/2011/binary/6dec11a.png

The disks that we formed before (rgrav-soft<0.5) via wind capture looked different, i.e. didn't look completely round and showed density substructure and complex motion. The new run looks promising.

Meeting Update 12.05

See Magnetized Clump project page (Scroll down to bottom) for updates.

Meeting Update 12/06/2011 - Eddie

Not much to say about the new cooling procedures. I'm waiting to hear back from Kris on that.

I have done quite a bit with the test suite. See ehansen12022011. I also updated all of the tests' wiki pages with Matt's new plugins.

First Version of Tests Plugin Completed

Current Functionality:

TestRefMacro — Returns the reference image of the specified test.

Example:

[[TestRef(TestName)]]

TestSimMacro — Returns the simulation image of the specified test.

Example:

[[TestSim(TestName)]]

TestLogMacro — Returns the run log of the specified test.

Example:

[[TestLog(TestName)]]

TestsMacro — Returns a table of reference and simulation images of all tests.

Example:

[[Tests]]

Test images may be manipulated using all standard arguments. For example:

[[Tests(120px)

Will rescale the width of all pictures to 120 pixels.

See example here: u/noyesma

The plugin reads available directories from the /data/tests directory on clover. If are are no corresponding images for a test in the SVN repository, the image entries will just appear as a question mark in the table. Hence, as new tests are added they will be automatically displayed in the Tests table.

CRL 618

We are trying finish these simulations. Bin and I have been experiencing some problems at bluehive and bluegene. These seem independent issues and I've open tickets 162 and 163 about them. We already have a reservation in bluegene for this Wednesday to solve the problem in that cluster.

As a plan B, I've sent 4amr versions of 2 of the runs to bluehive's queue.

5 dec 11. The 4amr level simulations, clump-stratifiedAMBIENT and jet-stratifiedAMBIENT, aborted after completing ~40% of the run. They were running in 24 afrank nodes at bluehive (vmem=19000mb). Chombos at that moment are ~ 700mb. They only report:

(A)

./launch.s: line 3: 5538 Killed ./astrobear

mpirun has exited due to process rank 18 with PID 5530 on node phy089 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here).

I tried restarting but they die shortly after, and produce the above error message. I will now recompile with check flags on, which will make the runs 10 times slower but should give more info at abortion.

About the 5amr runs and bluegene. We have a reservation in that cluster for this Wednesday to do debugging.

Several test suite updates (astrobear changeset 701 + 702)

I finished fixing the RadiativeInstability files so that they now only produce 1 chombo at the correct computational time. Then, I updated the reference chombos that are in the /data/tests directory on Clover. I also noticed some errors while doing the post processing on this problem. I fixed the bear2fix.data files in astrobear and the clover directory.

Another thing that I noticed while doing the chombo comparison for RadiativeInstability was a relative error of infinity! I realized that the y-momentum is zero for this problem. Therefore, I rewrote bear2fix's i_compare.f90 to avoid this divide by zero issue. It now replaces an infinite error with one of three things:

1) zero error if the new chombo also has zero momentum component

2) an error equal to the new chombo's momentum component if the new momentum component is very small (which would lead to an error within the 'successful' tolerance)

3) an error relative to the average of the reference chombo's momentum components (which could lead to a 'success' or 'fail')

Lastly, the OrbitingParticles problem was added to the test suite. Producing a useful image was the most difficult part here. I "tricked" bear2fix into plotting the sink particle potential. To do this I told bear2fix to plot By. The particle potential is stored in the same location in the q array that By would be if there were magnetic fields.

All the relevant aforementioned changes were pushed to scrambler_devel.

Rotating simul update

Check out my rotating BE sphere~!

http://www.pas.rochester.edu/~erica/sinks.html

Disks tilt

Regarding Keplarian disks, with gamma=1.001, which move about a point gravity without any disturbance from the ambient medium. I've found a correlation between the point gravity softening radii, rs, and the undesired disk gas motion. This is for the binary simulation setup (i.e. grid size, BC, and scales).

Basically, for 4 amr levels, disks are symmetric and stable (i.e. neither tilt nor explosion) for rs>0.5, but unstable for smaller rs. More exploration is needed, but thing are looking pretty good.

r=0.5 is about 3% of the grid size. The wind-capture disks that we've been seen so far had a radius close to 0.5, but his may change now with a different r_s. for 4 amr, r_s this was resolved with 8 cells

The stable disks I've been seeing have a radius of .5 too. This means that all the initial disk gas is in the flat part of the grav potential. I was using much smaller r_s because I wanted to model the grav potential as accurate as possible.