Meeting Update 09/08/2014
Over the past few weeks I've spent my time shuffling around data and organizing things to make visualization not… impossible? So now we have all of the data together in their respective directories, shared between bamboodata and grassdata. Although one can use Erica's directories in grassdata as "home base" as I made sym-links to the bamboodata directories. So everything seems like it is stored with its applicable files on grassdata.
Current Disk Usage:
/bamboodata/ | 86% |
/grassdata/ | 89% |
Now we're doing post processing on the data, along with resolution testing. We have updated our wikipage (see: CFRunStatsHighRes).
I am also studying for the physics GRE, and given my study/work schedule I might not show up for group meetings. I'll still post updates.
Running Jobs
- Currently:
- Bluestreak (BS):
- Post processing (PP) on our chombos (reservation, for B1S60 and B10S15 256 nodes). Once those two runs are done, we'll do the other two. Eventually we'll need to throw the NoShear case into the mix (see 2.).
- Convergence testing at AMR level 3 (pending in Standard queue for both shear 60 cases). Once these jobs are done, we'll do the other two if necessary.
- Stampede:
- Running the NoShear case on there. Have done approximately 3 restarts. Current restart began 187, so this should be the last restart there. After each restart completes and I restart the next one, I tranfer files back to our local directories. Once we have all the files, I'll transfer them to BS to do PP.
- Bluestreak (BS):
- Future:
- PP on BS of NoShear data.
- PP for B10S30 and B10S60 on BS.
- Convergence Tests for the rest of the data sets.
Tranferring Files
- Once all PP on BS is done, transfer everything back to local directories.
- Tranferring NoShear to BS and local machines.
Visualization
- Erica submitted the proposal to the Vista Collaboratory Contest this past Friday.
- Visualization on BS using x2go.
Meeting Update 08-05-2014
File Transfer from BS & Run Updates
- Went through all of the chombos on visit. All look great, no issues.
- Deleted all of the chombos on BS as I was over soft limit and could no longer write files.
- Been updating Erica's Production Runs page: https://astrobear.pas.rochester.edu/trac/wiki/u/erica/CFRunStatsHighRes
- Reservation starting Thursday at 8AM for 5 days. Plan on running B1S60 at 128 nodes, B10S15 at 256 nodes, and B10S30 at 128 nodes. Currently have B10S60 running in th standard queue, already at chombo/sink 157.
Post-Processing & Local Machines
- All of the files I have run on BS are transferred over either on grassdata or bamboodata (some b10s30 (143-153). Can check ProductionRuns directories under madams in grassdata/bamboodata.
- We have the following chombos: 144 (B10S15), 153 (B10S30), 153 (B10S60), and 130 (B1S60)
- We have about ¾ of all the runs done.
- Post-Processing is up to date for B1S60 (completed last night), currently doing post processing on both bamboo and grass (bamboodata) for B10S60 and B10S30 respectively.
Going Forward
- Reservations on BS. E-mailed Carl to see if he could take off the soft limit for Erica's account, including my own.
- Getting onto Stampede so we can split the jobs between the two machines.
- Need more space on our local machines. Many of our files are scattered between our directories. It would be smart to manage them in the same directories.
Meeting Update 07/28
Now in full control of Erica's CollidingFlows data on Bluestreak. Completed first reservation (5 days) today. Can see our progress here (CFRunStatsHighRes).
- E-mailed Carl about the soonest I can get another reservation (4-7 days on BS). (Update 14:47: Got our reservation extended to next Monday at 8am).
- Erica is working on what is necessary to get us up on Stampede.
Currently:
- scp-ing files from BS to /grassdata/madams/.
- Post-processing on Bamboo.
- Running B10S30 on our extended reservation on BS at 512 nodes for 48 hours (07-28-2014 14:49).
- Submitted B10S60 to the standard queue on BS at 128 nodes for 24 hours… pending, but we'll see what happens.
Some highlights from CFRunStatsHighRes Page:
Beta | Shear | Machine, Path | Frames Completed / Left | Filling Fracs | Walltime Taken / Left | (1-2) Info, (3) Message allocations | Framerate (hrs/frame), No. cores | Notes |
1 | 60 | BS, /scratch/madams15/CollidingFlows/Beta1/Shear60/Restart/ | 122/200 | 0.106 0.559 0.631 0.321 0.467 | 16.8days/0days | 282.5 gb, 197.6 mb, 64.0 mb | ~5hr/frame, 2048 cores | Reservation. Stayed with 128 nodes. Started with 48 hours left on reservation. |
10 | 15 | BS, /scratch/madams15/CollidingFlows/Beta10/Shear15/Restart | 139/200 | 0.386 0.698 0.616 0.496 0.468 | 22.1days/0days | 422.4 gb, 296.5 mb, 64.0 mb | 7hr/frame, 2048 cores | Reservation. Restarted with 128 nodes on 07/26. |
10 | 30 | BS, /scratch/madams15/CollidingFlows/Beta10/Shear30 | 128/200 | 0.136 0.587 0.601 0.465 0.418 | 1.2mo/0days | 377.0 gb, 252.4 mb, 64.0 mb | 9hr/frame, 2048 cores | Reservation. Started Jul 24 08:46, cancelled to test other runs, started again at Jul 26 16:41. |
10 | 60 | BS, /scratch/madams15/CollidingFlows/Beta10/Shear60/Restart/ | 151/200 | 0.180 0.547 0.544 0.318 0.269 | 6.7days/0days | 305.3 gb, 211.8 mb, 64.0 mb | ~4hr/frame, 2048 cores | Reservation. |
Going Forward:
No. Nodes | Case | frames/hr | frames/day | Chombos Left | Days to Complete | Weeks to Complete |
128 | B1S60 | 0.2 | 4.8 | 78 | 16 | 2.29 |
B10S15 | 0.143 | 3.5 | 61 | 17 | 2.43 | |
B10S30 | 0.111 | 2.67 | 72 | 27 | 3.9 | |
B10S60 | 0.25 | 6 | 49 | 8 | 1.14 |
No. Nodes | Case | frames/hr | frames/day | Chombos Left | Days to Complete | Weeks to Complete |
256 | B1S60 | 0.4 | 9.6 | 78 | 8.1 | 1.16 |
B10S15 | 0.29 | 6.96 | 61 | 8.7 | 1.24 | |
B10S30 | 0.22 | 5.28 | 72 | 13 | 1.86 | |
B10S60 | 0.5 | 12 | 49 | 4 | 0.57 |
No. Nodes | Case | frames/hr | frames/day | Chombos Left | Days to Complete | Weeks to Complete |
512 | B1S60 | 0.8 | 19.2 | 78 | 4 | 0.57 |
B10S15 | 0.57 | 13.68 | 61 | 4.5 | 0.643 | |
B10S30 | 0.44 | 10.56 | 72 | 7 | 1 | |
B10S60 | 1 | 24 | 49 | 2 | 0.29 |
- Depending on how quickly we can get onto Stampede, I am estimating that we'll be done with these in 2 weeks if we do this most efficiently.