My Roe solver + entropy fix is working now. Here is the latest updated page -

​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/RoeSolver#Thecode-EntropyFix

I am out of town until Wed., but plan on continuing my Toro tour de force until I get back.

Next step for the Roe solver is importing it into astrobear. Will ask around to see what this entails.

2.5D Runs

We are doing these runs at a higher resolution now. Approximately 3.5x the resolution used by Raga. I have tried running these on alfalfa (16 procs) and bluestreak (256 procs). Below is a summary of what is going on…

All the runs that I have had to restart on bluestreak quit with the same error. See the attached files/images for the standard output from the beta = 0.4 restart, and also the corresponding error messages from core 64.

3D Runs

I have been trying to do these runs on kraken on 480 procs…

There are no error messages from these runs. They just start taking forever because they keep requesting restarts. I attached the standard output from the restarted beta = 5 run.

UPDATE

Turning off refinement for B-field gradients helped the run on alfalfa get past frame 6.6, so I'm trying this on the other machines as well for the runs that have not yet completed.

Need a letter demonstrating the covering of trip expense to Germany by tomorrow.

TSF Checked one thing we discussed last week, the initial cloud mass is about 1.15 solar mass. In the Krumholz runs, the final mass of the star is about 0.95 solar mass as indicated in my previous blog post. This means the star takes about 82% of the initial cloud mass, 3% of the star mass is wind material, so about 80% of the initial cloud material ends up in the star.

Rotational TSF Got the rotating TSF set up and running. The parameter is similar to those defined in Banerjee:

What we have:

A1 is currently running. I think we should at least have A1 and O1 (preferably high res) finished before the conference. After that we can focus on the rest and writing the paper. MHD supported cloud likely require a different cloud setup so that should be another paper.

NLUF From the discussion this morning it seems to be able to simulate the local change of resistivity at least to a certain degree (analytic model), is important. What I got as expected parameters:

r_{clump} = 0.5 mm
\rho_{clump} = 50 mg/cc
\rho_{ambient} = 0.5 mg/cc
T_{ambient} = 300K
T_{postshock} = 10eV
v_{shock} = 100km/s
Material: Argon.

Three problems:

1. the preshock material is cold and less conductive, the postshock material has higher temperature thus more conductive, which is not simulated currently.
   
2. the conductivity inside the ball tends to be low because of low temperature, however, it is this field that is important when shocked. If the field inside the ball diffuse quickly into the ambient as a result of high resistivity, we may not get the desired behavior.
   
3. Whether to look down the axis or orthogonal to the axis.
   
   

This past week, I've been updating Martin's module to work with latest revision of the code. So far, we've gotten the following 3D runs:

(1) Jet, constant ambient
(2) Jet, stratified ambient
(3) Clump, constant ambient
(4) Clump, stratified ambient

ex:

• The torus ambient does not seem to be running properly at the moment, and needs some more work.
• We want to edit the module to run in 2.5D instead of 3D.
• Meeting with Martin tomorrow to discuss next steps

As far as reading, I've read:

Shaping Bipolar Planetary Nebulae: Effects of Stellar Rotation, Photoionization Heating, and Magnetic Fields (Garcia-Segura, et al. 1999)

Magnetically Driven Winds from Post-Asymptotic Giant Branch Stars: Solutions for High-Speed Winds and Extreme Collimation (Garcia-Segura, et al. 2004)

From Bipolar to Elliptical: simulating the morphological Evolution of Planetary Nebulae (Huarte-Espinosa, et al. 2012)

The Formation and Evolution of Wind-Capture Disks in Binary Systems (Huarte-Espinosa, et al. 2013)

Outflows from Evolved Stars: The Rapidly Changing Fingers of CRL-618 (Balick, et al. [submitted])

Exploring Model Paradigms for the Cores of Active Pre Planetary Nebulae (Balick, et al. [draft] )

Also picked up Toro for some supplementary reading this summer.

• Tickets
  1. new: #288(Krumholz accretion creates multiple particles when using particle refinement buffer)
  2. closed: none

• Users
  1. new one from Uppsala University Sweden asked for the code
  2. another meeting with LLE

• Wiki
  1. new latex plugin (stable version): the current way to write a latex equation is

[[latex($ $)]]

instead of

[[latex($ $)]]

or

{{{#Latex }}} 

I can do the transform for you if you have wiki pages with equations that couldn't show correctly

• Worked on
  1. 3D Colliding Jets: ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/bliu05242013

2. Single jets negative temperature:
   1) 0AMR runs to the end on 16 cores of bamboo:
   2) 16X160+2AMR on 16 cores of bamboo: ​http://www.pas.rochester.edu/~bliu/RAGA/m16X160_2AMR.gif
   3) 16X160+4AMR on 16 cores of bamboo: got negative Temperature at frame 3
   4) 16X160+4AMR on 1 core of bamboo: got negative temperature at frame 59
   5) working on tracing back a revision which worked for Eddie

• Boundary Effects Page for BE runs - ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/u/erica/BEboundary - I added to this page my current understanding of boundary problems with simulations, as well as what I am seeing in my sims. If anyone could help refine this understanding, will be appreciated! Thanks

• Roe Solver with Entropy Fix, is working on Tests 1 and 3, but not 4 and 5 yet -​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/RoeSolver

• Things currently on my plate: debug code, jeans instability sim, BE paper analysis.

Got many restart requesting due to the nans in flux before the Jets meet.

1. Jets meet at frame 30

2. movie: ​http://www.pas.rochester.edu/~bliu/PulsedJets/colliding.gif

3. freezes at 78, restart from 77 still freezes

    filling fractions   =   0.969  0.930
    Current efficiency  =  86% 
    Cell updates/second =       1990      4487  44%
    Wall Time Remaining =   151.1 kyr at frame   77.9 of    100
    AMR Speed-Up Factor =       0.1904E+04
        Advanced level  2 to tnext= 0.2234E+01 with dt= 0.2327E-11 CFL= 0.3892E-01 max speed= 0.4181E+10
        Advanced level  2 to tnext= 0.2234E+01 with dt= 0.2327E-11 CFL= 0.3892E-01 max speed= 0.4181E+10
      Advanced level  1 to tnext= 0.2234E+01 with dt= 0.4654E-11 CFL= 0.2994E-01 max speed= 0.3217E+10
        Advanced level  2 to tnext= 0.2234E+01 with dt= 0.2327E-11 CFL= 0.3892E-01 max speed= 0.4181E+10
        Advanced level  2 to tnext= 0.2234E+01 with dt= 0.2327E-11 CFL= 0.3892E-01 max speed= 0.4181E+10
      Advanced level  1 to tnext= 0.2234E+01 with dt= 0.4654E-11 CFL= 0.2998E-01 max speed= 0.3221E+10
    Advanced level  0 to tnext= 0.2234E+01 with dt= 0.9307E-11 CFL= 0.1964E-01 max speed= 0.2110E+10
    Info allocations    =     1.5 gb  130.4 mb
    message allocations =   ------     35.5 mb
    sweep allocations   =   ------     54.3 mb
    filling fractions   =   0.969  0.930
    Current efficiency  =  86% 
    Cell updates/second =       1990      4487  44%
    Wall Time Remaining =   136.9 kyr at frame   77.9 of    100
   

I am now:

1. reading an introduction to close binary stars, R. W. Hilditch
2. reading principles of astrophysical fluid dynamics, Clarke and Carswell
3. reading Riemann solver, Toro

and papers:)

Agenda

1. Review Tom Jones at al 1996:
   

​http://www.pas.rochester.edu/~shuleli/0521/34308.web.pdf

2. Rethink the sink particle creation. References:
   

​http://adsabs.harvard.edu/abs/2004ApJ...611..399K
​http://adsabs.harvard.edu/abs/2010ApJ...713..269F
​https://astrobear.pas.rochester.edu/trac/astrobear/blog/johannjc03062012

3. Look at more frames from my triggered collapse sim:
   

Full length movie:
​http://www.pas.rochester.edu/~shuleli/0521/m1k.gif

Star Mass (solar mass):


Accretion Rate (g/s):


Mixing Ratio (one in a thousand):

4. Review Erica's BE collapse problem.

• Compared astrobear with loop and sweep direction reordering
  • 10% speedup on 2D 16²+4
  • 20% speedup on 3D 64³+0
• Next going to look at ways of using shared memory efficiently.

Restarted the triggered star formation run after finding out the original run has created more than 1 zero-mass particles. After tweaking the code, it almost works now, at least for the AMR runs. The following is the Mach = 1.5 run with Krumholz accretion:


Wrote a small cpp program to strip the sink particle data as curve files and plot:


Interestingly, the mass of the star is very close to that of the initial cloud, I have not yet examined how much wind material is contained inside the star.
​http://www.pas.rochester.edu/~shuleli/0520/m1k.gif

Later I found some weird behaviors in the uniform collapse problem. Both the Fedderath accretion and Krumholz accretion creates more than particles:

​http://www.pas.rochester.edu/~shuleli/0520/ucf.gif

​http://www.pas.rochester.edu/~shuleli/0520/uck.gif

This is also observable if we turn off adaptive refining and use the buffered zone refining instead. The density distribution is slightly different:

Options are:
(1) Complete the AMR runs that looks good.
(2) Find what's wrong with buffered refining, and focus on high res runs.

• Wiki page on Chapter 10 (HLL and HLLC solvers) — ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/ApproxFlux

• Wrote a Roe solver without entropy fix, but results are wonky — ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/RoeSolver

• Making a poster for PPVI, here is a very rough first draft —

• Removed chaig and eschroe files from grassdata, but don't have permissions to remove yirak's

I was really busy last week, and out of town for a few days, so I didn't get as much done as I would have liked. Here's where I'm at right now…

• My 3D pulsed jet runs are having memory issues on both bluestreak and kraken. I ran valgrind but found no memory leaks, so I'm not sure why this is happening. I could probably get some lower resolution runs through, but I definitely can't resolve the cooling length if this doesn't get fixed. I made a couple of tickets so that Jonathan, Baowei, and I can work on this issue.

• I altered my jets module to handle colliding jets. I also implemented a switch to do co-rotating magnetic fields and counter-rotating fields. Here are some low resolution images and movies which show that my modifications worked:

movie​

movie​

• I didn't get any paper writing done last week, but I should have enough free time this week to make some progress.

• Users
  1. Met with LLE: Most of time were spent discussion the results of ablative RT results. Arijit shows the results with perturbation from Betti's code. Rui shows the results of ablation balance from his 3D code. Shule will summarize what he's been doing.
  2. Very positive Feedback from Josh of Clemson.
  3. Wrote to a user in China asking for feedback, no reply yet
  4. The Clemson and LLE users asked computational scientist/system administrator to install AstroBEAR on their machines. Review my instruction for users to install

To install and run it, you will need a Linux system, MPI, and libraries like fftw3, hdf5 and maybe hypre. You may find more details on our wiki page:
https://astrobear.pas.rochester.edu/trac/astrobear/wiki/UserGuide

Edit Makefile.inc. Sounds too complicated for common Linux guy? Considering it usually takes me about 2 hours to install all the libraries required & the code to a new system, and sometimes I need to ask system admin's help for job submission, there might be a big barrier for new users to pass before they know how good our code is. Simpler&easier-installation version? Make configure file high priority?

5. Got Zhuo accounts and key to the office. Walked him through the whole process of getting, compiling and running the code on local machines

• Tickets
  1. New: #286(Memory Allocation Error on BlueStreak), #287(Virtual Memory Error on Kraken)
  2. Closed: none

• Machines & More disk space
  1. Move alfalfa & bamboo to 476?
  2. suggestions from Dave&Rich for disk space: ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/bliu05152013

• Worked on optimization (#285)
  1. switch all the FORALLs to distributed DO LOOPs in sweep_scheme.f90, no optimization found
  2. results are shown here: ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/bliu05132013_2

Dave & Rich use Hard drive docks which might be a good solution for our current space shortage. Here's some thoughts:

1. They are very cheap comparing other options. And it's very easy to expand the storage. <300 to get the dock and 4TB harddrive(the maximum individual the dock supports) which is a little less than grass or alfalfa. We can get 12TB space by 500.

2. We can archive the data to these harddrives once the paper published. We can create a folder on our local machines for the data to be archived. We will have a file keeping records what data are moved in the folder and to be archived. Whoever moves data to the folder will be responsible for putting those records to the file. Every three/four months, I will archive the folder to the hard drive and clean the whole folder.

A brief wiki page on the subject here - ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/ApproxFlux ' Next is learning about the ROE solver, which seems to be based on the HLLC paradigm.

So it wasn't too difficult to modify my pulsed jet module to handle colliding jets. I did a quick low-resolution, hydro run to test out my modifications, and it looks pretty cool.

Density	Emission Map
movie​	movie​

Next, I have to edit the magnetic field part, but this shouldn't be too difficult either.

Replace the vectorized FORALL loop with parallelized DO loops in sweep_scheme.f90 An example is to replace:

        DO i=mB(1,1), mB(1,2)
               FORALL(j=mB(2,1):mB(2,2),k=mB(3,1):mB(3,2))
                  beforesweepstep_%data(beforesweepstep_%x(i),j,k,1,1:NrHydroVars) = &
                       Info%q(index+i,j,k,1:NrHydroVars)
               END FORALL
            END DO

by

    !$OMP PARALLEL DO PRIVATE(k,j,i) COLLAPSE(3)
            DO k=mB(3,1),mB(3,2)
               DO j=mB(2,1),mB(2,2)
                  DO i=mB(1,1), mB(1,2)
                     beforesweepstep_%data(1:NrHydroVars,1,i,j,beforesweepstep_%x(k)) = Info%q(i,j,index+k,1:NrHydroVars)
                  END DO
               END DO
            END DO
                !$OMP END PARALLEL DO

Testing results on Blue Streak are

1. 128^{3 + 4AMR, Current Revision Running Time on 512 cores: 224.57 (Tasks per node=16)}

Tasks per node	OMP_NUM_THREADS	Total Running Time
1	32	3375.17
2	16	2019.94
4	8	1265.58
8	4	1052.74
16	2	907.62
32	1	1151.02

Tasks per node	OMP_NUM_THREADS	Total Running Time
1	64	>3600
2	32	2039.45
4	16	1181.27
8	8	946.2
16	4	741.81
32	2	737.68
64	1	877.07

2. 32^{3 + 4 AMR, Current Revision Running Time on 512 cores: 33.26 (Tasks per node=16)}

Tasks per node	OMP_NUM_THREADS	Total Running Time
1	64	191.42
2	32	122.68
4	16	82.43
8	8	70.78
16	4	72.78
32	2	85.65
64	1	129.95

Tasks per node	OMP_NUM_THREADS	Total Running Time
1	32	164.59
2	16	105.90
4	8	86.67
8	4	79.62
16	2	84.98
32	1	128.47

The job submission script on Blue Streak is like

#!/bin/bash
#SBATCH -J strongTest
#SBATCH --nodes=32 
#SBATCH --ntasks-per-node=4
#SBATCH -p debug 
#SBATCH -t 01:00:00

module purge
module load mpi-xl
module load hdf5-1.8.8-MPI-XL
module load fftw-3.3.2-MPI-XL
module load hypre-2.8.0b-MPI-XL

ulimit -s unlimited
export OMP_NUM_THREADS=16
#1node 8 processors
srun astrobear > strong_4ThreadsperNode_X16.log
                                      

swap the DO loop layers to i, j, k — the difference of running time is small comparing k,j,i case

                !$OMP PARALLEL DO PRIVATE(i,j,k) COLLAPSE(3)
            DO i=mB(1,1), mB(1,2)
               DO j=mB(2,1),mB(2,2)
                  DO k=mB(3,1),mB(3,2)
                     beforesweepstep_%data(1:NrHydroVars,1,i,j,beforesweepstep_%x(k)) = Info%q(i,j,index+k,1:NrHydroVars)
                  END DO
               END DO
            END DO
                !$OMP END PARALLEL DO

Tasks per node	OMP_NUM_THREADS	Total Running Time
1	16	>3600
2	16	2099.57
4	16	1208.53
8	8	912.56
16	4	758.78
16	2	969.74
16	1	1436.98

• Tickets
  1. new: none
  2. closed: #283(compiler on Kraken complains about long lines), #284(trouble submitting jobs on bluestreak)

• Users:
  1. will meet with Ruka and set accounts for him

• CIRC poster session
  1. ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/bliu05102013

• Worked
  1. #285 (Optimize AstroBEAR 2.0 on Blue gene/Q)

• Read Toro chapter 9, notes here - ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/ApproximateRS

• Almost through Toro chapter 10

• Astrobear movie is done (or near done?)

• 3D pulsed jets are working…ehansen05102013. I have a set of runs waiting to go on blue streak.

Paper Outline

• Abstract
  • general overview of paper

1. Introduction
   • mention observations
   • motivation for this research
   • what questions does this paper address:
     • simulations such as these have been done before, but only for H-alpha emission, and only in 2.5D…what new information do we obtain by also looking at [S II] emission?
     • also, what new information do we obtain by extending the simulations to 3D? are "nose-cones" still present?
     • how does magnetic field strength affect the clumps and emission within a pulsed jet?
     • can we use H-alpha and [S II] emission to estimate magnetic field strength in YSO jets?
2. Numberical Simulations
   • Methods
     • equations being solved with astrobear and type of cooling
   • 2.5D
     • define the problem set-up and parameters
   • 3D
     • more brief description of problem set-up
3. Results
   • 2.5D
     • density plot, a few lineouts (rho, v, T), emission map
   • 3D
     • density slice, a few oineouts of the slice (rho, v, T), emission map
     • 3D image
4. Discussion
   • more qualitative than quantitative
   • differences between 2.5D and 3D
5. Conclusions
   • summarize the paper
   • relate simulations back to observations
   • lingering questions and future work
     • are we likely to learn anything new from simulating other magnetic field geometries?
     • also, is there anything else to learn from looking at other emission lines such as [N II] or [O I]?

• References

Some pictures for this year's CIRC poster session. Thanks for participating.

So it was a long and annoying task, but Jonathan and I finally got it working. Most of the changes I had made to my module to go from 2.5D to 3D were correct. The two biggest changes that were made today which appeared to make it work were:

1. Doubling the problem domain. Now simulating half the jet instead of a quarter.
2. Switched interpolation order from 3 to 2.

Jonathan suspects there might be a bug in the 3rd order interpolation scheme which caused assymetries in the z-direction in my simulations. I'm going to run a couple more tests and post a ticket on this.

Here is a low resolution movie of the H-alpha and [S II] emission map from a beta = 5 run.

movie​

Most of last week I was working on my graph spec project.
​http://www.pas.rochester.edu/~shuleli/0331/magneticbeta.png
​http://www.pas.rochester.edu/~shuleli/0331/magneticbeta2.png
​http://www.pas.rochester.edu/~shuleli/0331/magneticbeta3.png
I will be working on the NLUF project, new calculations based on the previous simulations (magnetic beta amplification vs compression ratio) to check the new design.

• Star Formation Jamboree was fun even though there were no talks, and hardly even any mentioning, of jets . Nevertheless, I had a good time talking with Fred Adams, Ralph Pudritz, and many other people.

• Made a poster yesterday for the CIRC Poster Session, so come by Goergen tomorrow anytime from 10-12 and VOTE FOR MY POSTER! It is by no means perfect, but it will suffice for now. I will probably use a different, but similar, one for PPVI, especially since I will have 3D runs before then.

• The 3D runs are still producing infinities. I've tried different things to fix it, but nothing has worked yet. I think with Jonathan and/or Martin's help, this should be working by tomorrow if not by the end of today.

• For the 1D runs, we are now just waiting to hear back from Pat. I'm gonna go ahead and do some analysis on the emission lines to see if I can better quantify the effect of the B-field.

*Fred Adams suggested non-zero velocity initial condition for collapse problem:

• there are semi-analytic solutions for the density distribution
• forms something 'like' a BE sphere
• suspects compression wave won't be set up at boundary
• more realistic then a hydrostatic clump, because he said you wouldn't form a clump in HSE with no motions — not sure about this point
• he would be interested in seeing these results
• talked about filaments feeding into clusters, on scales ~2pc from cluster that is ~2pc across

*Ralph Pudritz said he's been working on:

• making a 'super cluster' sink particle that randomly samples an IMF, produces radiation to provide feedback to parent cloud
• radiation modeled by using a combination of ray tracing and flux limited diffusion to resolve feedback on scales from GMC down to large clumps
• recently modeled the formation of GMC's from gravitational instabilities in galactic disk, & has online database of these clouds and their various properties
• uses Flash

*Hennebelle collaborator (don't remember name) works with BE spheres in turbulent, magnetized environments

• CIRC poster session
  1. Time: start at 10 a.m. — 9:45 am if you have a poster
  2. Location: Goergen Hall

• Users
  1. New: from United States Navel Academy (Interested in learning more about astrophysics, including possible future involvement in mass wave detection devices such as LIGO and LISA, with the hope of performing and enhancing universe dynamics simulation techniques.) and Xiamen University (Astrophysical Simulations)

• Tickets
  1. New: none
  2. Closed: none

• Worked on
  1. scaling test of AstroBEAR on the whole machine of BlueStreak
  2. poster for CIRC poster session: ​http://www.pas.rochester.edu/~bliu/Posters/poster_circ.pdf
  3. code optimization with OpenMP. (#285) testing it on Stampede.

• I have put together a near compendium on learning the Godunov method, my results, and discussion here - ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica/GudonovMethodEuler

• I have learned A LOT, see previous post also — ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/erica05032013

• I am scheduled to shoot interview for astrobear movie on Friday.

• Conference went well, here are some notes I took on my discussion with Prof. Adams and Pudritz - ​https://astrobear.pas.rochester.edu/trac/astrobear/blog/erica05092013key

• Will begin next steps on my summer numerics schedule now

Happy to say the bug(s) have all been tracked down and eradicated. I posted results for test 4 on my wiki here — ​https://astrobear.pas.rochester.edu/trac/astrobear/wiki/u/erica

But the page is still under construction. Would like to take another day to type up my findings and chew over the results.

I'd also like to say… Running my code produces standard out like astrobear… printing the iterations over cells and over time. Running the program takes a decent minute to get through all the calculations, through all the loops and such. Very neat. I can see computational cost already; it is impressive to imagine how fast the cost must grow by adding more dimensions, grids, and physics, like astrobear.