I ran the Beta 10 No Shear case on Stampede with 1024 cores. Here is the result (see Table below):

\overline{t_{0}} = 87.67 \rightarrow r_{0} = \frac{1,440 \text{min}}{\overline{t_{0}}} = 16.43 \text{ frames a day}

So if we're at frame 246, we have 154 frames left. So dividing 154 by our rate, we have 9.4 days (225.6 hrs) to run this simulation out. Thus, 225.6 * 1024 = 231,014.4 cpu hrs. Multiply this by 4, as we have 4 runs, yields approximately 924,057.6 cpu hrs total. This is not much different then the total result from ​last week. It does not seem economical to run these on 1024 cores; in my opinion we might as well just run these on 2048 cores as they'll be faster but have little to no shift in cpu hrs.

Perhaps we should choose just a few cases on 2048 cores?

If on 2048 cores we estimate 34.85 frames a day (average of rates from last blog post) with approximately 164 frames left (average from last blog post) that implies that we have approximately 5 days to run a simulation, or 113 hours. This is approximately 231,304 cpu hrs. With 3 runs, that is 693,911 cpu hrs. With 2 runs that is 462,607 cpu hrs.

Perhaps we could split the runs between machines? However we aimed to use Stampede because it is so fast in comparison to the likes of BlueStreak.

Table. The Beta 10 Shear 0 run with current frame for which the hours were sampled and averaged to do the brief calculations above.

Running the CollidingFlows problem out from frame 200 to 400 to double the time and see if we can observe any more sink formation. Given that this run is really computationally intensive, I've done a quick calculation for cpu hrs based on some current runs I am doing on Stampede. All runs are in the normal queue for 24 hrs on 2048 cores. The table below provides the current frame number at which I collected this data. We can see that the average time for our code to spit out a frame is (underscores correspond to the run):

\overline{t_{0}} = 44.\bar{3} \text{min}

\overline{t_{15}} = 46 \text{min}

\overline{t_{30}} = 47 \text{min}

\overline{t_{60}} = 32 \text{min}

Given that we have 1,440 minutes in a day, implying that we'd spit out the following frames per day:

r_{0} = \frac{1,440 \text{min}}{\overline{t_{0}}} \approx 32.5 \text{ frames per day}

r_{15} = \frac{1,440 \text{min}}{\overline{t_{15}}} \approx 31.3 \text{ frames per day}

r_{30} = \frac{1,440 \text{min}}{\overline{t_{30}}} \approx 30.6 \text{ frames per day}

r_{60} = \frac{1,440 \text{min}}{\overline{t_{60}}} \approx 45 \text{ frames per day}

Considering that the difference between the current frame and the last frame (400) for beta10 shear 0, 15, 30 and 60 respectively are 179, 182, 159, and 136, we're looking at running these out for approximately 5-6 days on 2048 cores. Specifically for b10s0: 5.5 days, b10s15: 5.8 days, b10s30: 5.2 days, and b10s60: 3 days. Using this number of days, that there are 24 hours in a day and we'd run these on 2048 cores, this puts us at a total of: 957,973 cpu hrs. THAT IS INSANE.

After a quick discussion with Erica and Baowei I've come up with the following short term plan: Once these jobs stop later today, I'll submit 1 job to the normal queue on 1,000 cores. For this run I'll make the same calculation and see if it is more economical when multiplied by 4. Baowei has also suggested to throw runs on Gordon, another machine owned by the Texans. We have a lot of SUs there, so he is currently setting me up. We currently only have 1,551,296 SUs available on Stampede — so running our jobs for this problem there could be quite precarious.

Table. Each run with current frame for which the hours were sampled and averaged to do the brief calculations above.

Over the past few weeks I've spent my time shuffling around data and organizing things to make visualization not… impossible? So now we have all of the data together in their respective directories, shared between bamboodata and grassdata. Although one can use Erica's directories in grassdata as "home base" as I made sym-links to the bamboodata directories. So everything seems like it is stored with its applicable files on grassdata.

Current Disk Usage:

Now we're doing post processing on the data, along with resolution testing. We have updated our wikipage (see: ​CFRunStatsHighRes).

I am also studying for the physics GRE, and given my study/work schedule I might not show up for group meetings. I'll still post updates.

Running Jobs

• Currently:
  1. Bluestreak (BS):
     • Post processing (PP) on our chombos (reservation, for B1S60 and B10S15 256 nodes). Once those two runs are done, we'll do the other two. Eventually we'll need to throw the NoShear case into the mix (see 2.).
     • Convergence testing at AMR level 3 (pending in Standard queue for both shear 60 cases). Once these jobs are done, we'll do the other two if necessary.
  2. Stampede:
     • Running the NoShear case on there. Have done approximately 3 restarts. Current restart began 187, so this should be the last restart there. After each restart completes and I restart the next one, I tranfer files back to our local directories. Once we have all the files, I'll transfer them to BS to do PP.

• Future:
  1. PP on BS of NoShear data.
  2. PP for B10S30 and B10S60 on BS.
  3. Convergence Tests for the rest of the data sets.

Tranferring Files

• Once all PP on BS is done, transfer everything back to local directories.
• Tranferring NoShear to BS and local machines.

Visualization

• Erica submitted the proposal to the Vista Collaboratory Contest this past Friday.
• Visualization on BS using x2go.

Noticed as I am scp-ing data on the CollidingFlows runs that some of the typical machines I used are starting to become full. I also do visualizations on Clover or my own machine, however they can be quite hard on my own graphics card. This has led to talk to Baowei and others about potentially looking into developing a new machine quote (see: ​https://astrobear.pas.rochester.edu/trac/blog/bliu02072012).

Given that the quote is 6 months old, that the price is not the same anymore. It has been suggested to me that someone in the group should e-mail Sean Kesavan <​sean@asacomputers.com> and ask him to give up an updated quote or change it to meet our requirements (referencing the quote number on the quotation). I also think these are the specs for Bamboo (​https://www.pas.rochester.edu/~bliu/ComputingResources/ASA_Computers.pdf).

In my opinion it might be a good idea to look into more disk space/storage and also get a new machine for visualization. In the mean time I took a look at our local machines and who is using them.

Commands I am using:

*du -sh * 2>&1 | grep -v "du: cannot" prints the disk spaced used for each user by printing standard error to standard out.

*df -h inside of the directory to see disk spaced used (along with %) and space that is available.

alfalfadata

The Disk Space Used

The Disk Space Available

bamboodata

The Disk Space Used

The Disk Space Available

botwindata

The Disk Space Used

The Disk Space Available

cloverdata

The Disk Space Used

The Disk Space Available

grassdata

The Disk Space Used

The Disk Space Available