Code Meeting Update 03/25/2014 - Eddie

Pulsed Jet Restarts

My latest run is just finishing on Kraken. It made it to 30 frames in 24 hours on 960 cores. There were 10 restarts triggered, 9 of which were high CFLs. Only one of them was the restart due to nan in flux. It looks like one of the toughest points was going from frame 24 to 25 which had 2 restarts and took almost 4 hours, but it eventually made it through.

I expect the frame output rate to slow down over time since the jet propagates into the grid and triggers more refinement. But the restarts make the run go even slower. It is possible that I could get this run to finish as is, given enough cpu hours, but I'm not sure how much we want to invest.

Right now, a good frame step that has no restarts, takes about 1.5 hrs. In the beginning, they only took about 10 minutes. If I assume that this trend will continue linearly, then at best the rest of the simulation would take 220 hours which is a little more than 9 days. With this many cores, this is 211,200 SUs. This is a best case scenario estimate. These numbers will increase due to triggered restarts, the need to physically restart the code every 24 hours will lead to inefficiency, and also the frame output rate probably decreases faster than linearly.

I have not yet had a chance to analyze the chombos from this run, but I attached the standard output to this post.


Czarships

Testing

We are still in the process of completing the new test suite which no longer uses bear2fix. I believe all the routines are in place, and now it is just a matter of setting up each of the problem modules to be compatible with these new routines.

Local Resources

Visit has been working very slowly on my machine (Grass). If I work on clover, ssh to grass, and then run visit, it is much faster. So I'm guessing that the bottleneck is on Grass' graphics card, and thus it has difficulty displaying things locally.

I don't know if this is something Rich can fix, but I think it might also be time for a new machine. Can we get the ball rolling on figuring out what we want and purchasing it?

Attachments (1)

Download all attachments as: .zip

Comments

No comments.