Czarships/ExternalResources/RenewCollidingFlow: XSEDE_proposal_BL_04102014.tex

 %\documentclass{article}
\documentclass[preprint2]{aastex}

\shorttitle{Study of Colliding Flows and Feedback in Star Formation}
\shortauthors{Frank, Carroll-Nellenback, Liu, Huarte-Espinosa, Li, Kaminski, Hansen}
\providecommand{\e}[1]{\ensuremath{\times 10^{#1}}}

\usepackage{color}
\usepackage{ctable}

\begin{document}
\title{{\color{red} Same Title???}Study of Colliding Flows and Feedback in Star Formation}
\author{Adam Frank, Jonathan J. Carroll-Nellenback, Baowei Liu, Mart\'{i}n Huarte-Espinosa,\\ Shule Li, Erica L. Kaminski,Edward Hansen}
%\usepackage{amssymb,amsmath}
%\usepackage{verbatim}
%\usepackage{graphicx}
%\usepackage[utf8]{inputenc}
%\usepackage[T1]{fontenc}
%\usepackage[square, comma, numbers]{natbib}


%\begin{document}

%\title{Study of Colliding Flows and Feedback in Star Formation}
%\author{Adam Frank, Jonathan J. Carroll-Nellenback, Baowei Liu, \\Mart\'{i}n Huarte-Espinosa, Jason Nordhaus}

%\maketitle

\begin{abstract}

{\color{red} The following two paragraphs need to be updated with TSF stuff}


[--- 

Observational evidence from local star-forming regions mandates that star formation occurs shortly after, or
even during, molecular cloud formation. Models of the formation of molecular clouds in large-scale converging
flows have identified the physical mechanisms driving the necessary rapid fragmentation. They also point to global 
gravitational collapse driving supersonic turbulence in molecular clouds. Previous cloud formation models
have focused on turbulence generation, gravitational collapse, magnetic fields, and feedback. In this work we propose exploring
the effect of structure in the flow on the resulting clouds and the ensuing gravitational collapse. We  seek to compare two
situations with different flow properties and processes (flow shear, geometry, magnetic fields, feedback). 
We have already found that structured inflows lead to a delay of local gravitational collapse (``star formation").
Thus, more gas has time to accumulate, eventually leading to a strong global collapse, and thus to a 
high star formation rate. Uniform inflows fragment hydrodynamically early on, leading to the rapid 
onset of local gravitational collapse and an overall low sink formation rate. 

Based on these results obtained with our previous XSEDE allocations (TG-AST120029) we propose to to carry out a series of three-dimensional Adaptive Mesh Refinement (AMR) parallel numerical simulations to study the cloud collapse and cluster formation under various conditions. This will include colliding flows with different shear angles, different magnetic field strengths and orientation angles with respect to the flow. 

---]

We are in full production mode with AstroBEAR, but the powerful numerical resources of the XSEDE that we are requesting are needed to make further progress on unraveling the mysteries of feedback process in star formation. To achieve this exciting goal we request the support of {\color{green} 4.0 million SU's} on Stampede at TACC, {\color{green} 2.1 million SU's} on Gordon at SDSC and {\color{green} 2.0 million SU's} on SuperMIC at LSU.
\bigskip
\end{abstract}



\section{Introduction} 

{\color{red} This section needs to be updated with TSF!!}

[----

The concept of flow-driven cloud formation \citep{Vazquez-Semadeni1995,Ballesteros-Paredes1999,Hartmann2001} can explain
two observational constraints on how molecular clouds form stars, derived from local star-forming regions: first, 
all local molecular clouds form stars, and second, the stellar age spreads are on the order of $1-2$~Myr, 
several times shorter than cloud crossing times 
\citep[see summary in ] {Hartmann2001,Ballesteros-ParedesHartmann2007}. 
The first constraint suggests that star formation sets in immediately (or even during) molecular cloud formation, 
and that the second constraint is trivially fulfilled in a scenario where the clouds themselves form in large-scale 
``converging" flows. The immediate (``rapid") onset of star formation in the forming clouds and the fact that the 
star formation efficiency is only a few percent \citep{Evansetal2009} mandates that the clouds are highly structured: 
local collapse must set in before global collapse can overwhelm the dynamics. 


The notion of cloud formation in converging flows has led to a series of numerical experiments investigating the 
physical processes relevant for the rapid fragmentation and for the control of the star formation efficiency. 
There is agreement across the models (despite different initial and boundary conditions) on the following results: 
(1) Rapid fragmentation is induced by strong radiative losses during the flow collision (possibly even by thermal 
instability if the clouds form from atomic gas), and by dynamical instabilities 
\citep{Hueckstaedt2003,Audit2005,Vazquez-Semadeni2006,Heitsch2008b}
(2) Turbulence in molecular clouds is a natural result of the dynamical instabilities during 
the cloud formation, and is driven by global gravitational collapse at later stages of the cloud evolution 
\citep{Vazquez-Semadeni2007}. Some evidence for this is seen in the filamentary nature of some clouds (Fig 1). (3) Strong, non-linear density contrasts can also be driven by 
self-gravity in finite clouds, due to geometry (or ``edge") effects \citep{BurkertHartmann2004}. 
(4) Although the rapid fragmentation can keep the star formation efficiency low, eventually, feedback or cloud 
dispersal is needed to prevent a large percentage of the gas to participate in gravitational collapse 
\citep{Vazquez-Semadeni2010}.

The most obvious difference in the simulation results is the morphology of the forming clouds. All models use some 
mechanism to break the symmetry in the colliding flows -- otherwise, the flow collision would result in a 
plane-parallel shock. Models with small-scale perturbations (``noise") in the velocities tend to lead to extremely 
flattened clouds with a strong ring due to the gravitational edge effect 
\citep{BurkertHartmann2004,HartmannBurkert2007} in finite, sheet-like clouds. If 
the velocity perturbations are imposed on larger scales (e.g as a turbulent power spectrum), or 
if the collision interface between the two flows is perturbed, turbulent structures form that are not 
necessarily coherent when viewed from different directions \citep{Heitsch2009}. 

\section{Previous Accomplishments}
{\color{red} new section required for the renewal !! }

\subsection{Summary of Scientific Discoveries}
{\color{red} new section required for the renewal !! }

\subsection{Details of Previous Accomplishments}
{\color{red} new section for the renewal !! }

\section{Accomplishments and the Way Forward} 

{\color{red} From the old proposal, need update with TSF or tailor to the section above}

\begin{figure}[ht]
\includegraphics[width=\columnwidth]{Panels.png}
\caption{Comparison of ``Clumpy'' run (top) and ``Smooth'' run (bottom) showing the projected column density along the flow direction (right) and perpendicular to the flow direction (left)}
\label{fig:panels}
\end{figure}

To understand better the effect of initial conditions on the clouds forming in the flow collisions, we have already completed a study \citep{Carrolletal13} comparing two (otherwise identical) cloud formation simulations, one with a smooth inflow, and one with a clumpy 
inflow of identical mass flux. Our studies were motivated by the analysis of \citet{Pringle2001}, suggesting 
that cloud formation out of warm atomic gas would require time and length scales too large to be feasible (see also \citet{McKeeOstriker}). This problem is solved by the realization that the flow collision is three-dimensional, 
allowing gravitational collapse and accretion along the dimensions perpendicular to the flows, and thus circumventing 
the one-dimensional limit on column density accumulation \citep{Heitsch2008a}. Yet, \citet{Pringle2001} suggest that clumpy  flows 
could reduce the accumulation and molecule formation time scale (especially if the clumps are already molecular). 
%In this proposal, we will study how hat effects of clumpy flow and smooth flows under a host of conditions including the presence of magnetic fields.

%In our previous work \citep{Carrolletal13} we presented the results of two simulations flow-driven molecular cloud formation. 
The models were identical
except for the physical conditions of the inflowing gas. One model (``Smooth") began with a completely uniform
flow. The other model (``Clumpy") used the same mass inflow rate, but with the gas distributed in dense clumplets with a filling factor of $5$\%.  In both cases, the thermal and dynamical fragmentation of the shocked collision region leads to turbulence, at
a level of $10$ to $20$\% of the inflowing kinetic energy. This is consistent with earlier results. The Clumpy
run showed a somewhat higher velocity dispersion initially, since the clumps are less easily stopped in a 
flow collision.(Fig 2) %(see \S\ref{morphologies}). (Fig 2)


Due to the lower compression factor in the Clumpy model, less gas is being cooled to higher densities than in the
Smooth run. Thus, the kinetic energy of the inflow is less efficiently dissipated. Together with a non-contiguous
distribution of cold, sub-jeans, fragments, this leads to a suppression of local collapse for nearly $20$~Myr after
the initial flow collision. At that point, sufficient mass has assembled to induce global collapse of the whole 
region, resulting in a ``star burst" (more appropriately, ``sink burst") at a high sink formation rate.
In contrast, the Smooth run shows local collapse already after $10$~Myr, at less than half the sink formation
rate of the Clumpy run. Due to the local nature of the thermal fragmentation, more fragmentation will occur
with increasing resolution \citep{Hennebelle2007}, thus, the times quoted here are {\em upper} limits for the onset
of local collapse. Nevertheless, structured flows can delay the onset of (substantial) local collapse.
Global collapse is only clearly visible in the Clumpy run.

The differences between Clumpy and Smooth inflows extend to the mixing efficiencies. Somewhat counter to 
a naive expectation, the Smooth initial conditions result in less well mixed material (and cores). This is
primarily due to the Nonlinear Thin Shell Instability (NTSI) funneling material preferentially into the troughs located far into the opposing
inflow. For the Clumpy run, the global collapse of the accumulated clumps erases all memory of the initial
inflow direction.

In \citep{Carrolletal13} we chose two extremes as our initial conditions. It is more likely that the inflows themselves
will contain turbulent velocity and density structures that are coherent in space. Spatial coherence leads to
stronger shocks in the collision region, and thus to more efficient energy dissipation. In that sense, our Clumpy
run is overestimating the effect of structured inflows.  In addition magnetic fields are likely to play a strong role in shaping the evolution of the flows.  The challenge ahead is to use a wider array of initial conditions and include more physical processes (such as magnetic fields) to explore the evolution of colliding flows and compare predictions from these cases with observations


%\begin{figure}[t!]
%\begin{center}
%\includegraphics[width=.99\columnwidth]{3DSimulation.png}%{3DSimulations of clumpy and smooth flows.png}
%\includegraphics[bb=0 0 511 512]{3DSimulation.png}%{3DSimulations of clumpy and smooth flows.png}
%\caption{This may not be the one. Jonathan need caption}
%\label{}
%\end{center}
%\end{figure}

\section{Research Objectives}

{\color{red} the follow is from the old proposal Need to be updated !!!}
We will focus on simulations and observational implications of colliding flows under a variety of conditions. This will include (1) colliding flows interacting with various degrees of shear and (2) colliding flows with different magnetic field strengths (3) colliding flows with different magnetic field orientations. Our computations will be carried out using AstroBEAR.

\begin{figure}[t!]
\includegraphics[width=\columnwidth]{AMR2.png}
\caption{Slice showing log density from 3D Colliding Flows simulation performed with AstroBEAR.  Also shown is the AMR mesh which contains 4 additional levels of refinement}
\label{fig:collapse}
\end{figure}

\section{Computational Approach} 
{\color{green} Updated by Baowei 4/10/2014!!}
AstroBEAR is an Adaptive Mesh Refinement (AMR), multi-physics code for astrophysics. AMR remains at the cutting edge of computational astrophysics. AMR simulations adaptively change resolution within a computational domain to ensure that the most important features of the dynamics are simulated with highest accuracy. By allowing quiescent regions to evolve with low resolution, AMR simulations achieve order of magnitude increases in computational speed. After a decade of development only a handful of AMR-MHD codes exist for astrophysics: (e.g. FLASH, ENZO RAMSES, ORION, CASTRO).

The UR astrophysics group successfully constructed and tested AstroBEAR, a fully parallelized, multi-dimensional AMR MHD code. The success of this effort is evidenced both in the code's completion (Cunningham et al 2009) and the papers published using AstroBEAR as it was developed through its radiation-hydrodynamic and MHD versions (a partial list includes: Poludnenko et al 2004ab; Cunningham et al 2005; 2006ab, Hartigan et al 2007, Dennis et al 2008, Yirak 2009, 2010, Li et al 2012, Huarte-Espinosa et al 2012).

The multi-physics capabilities of AstroBEAR have been significantly expanded by including solvers for elliptic and parabolic equations. Adapting the linear system solver HYPRE, we now routinely simulate systems in which self-gravity, heat conduction and magnetic resistivity are important. Radiation transfer in the diffusive limit is currently being added. In addition, AstroBEAR can treat gravitationally interacting point particles which accrete mass. 

\subsection{AstroBEAR Scaling}\label{subsec:scaling}
AstroBEAR is designed for 2D and 3D adaptive mesh refinement (AMR) simulations which require algorithms that are highly parallelized and manage memory efficiently. AstroBEAR uses a hierarchical approach to parallelization suitable for multicore architectures in which large-scale patches of data are distributed to nodes using MPI and the work for an individual patch is distributed across the cores on a node using OpenMP directives. AstroBEAR also employs new techniques such as load balancing by threading the grid advances on each level with preference going to the finer level grids. 

Here we present strong scaling and weak scaling results for AstroBEAR. In Figure \ref{fig:scalingPlot}, we report scaling test results on Stampede at TACC. The resolution we used for these test are $40^{3} + 5$ level AMR which is same as the computation we are planning to do. The strong scaling test plot of the current code shows a slope $-0.83$ on up to 1024 cores and $-0.69$ on up to 4096 cores while the slope for perfect scaling is $-1$. This shows AstroBEAR has an excellent scaling on Stampede. For the weak testing results, we keep the work load on each core same (each has $64^3$ cells). So the weak scaling result measures the efficiency of data communication between the processors. Our test results show the communication efficiency drops only $15\%$ running on $1024$ cores and $27\%$ on $4096$ cores.  More results and details about the performance of AstroBEAR can be found in the report of "Code Performance of AstroBEAR2.0".
\begin{figure}[t!]
\centering
%\subfigure[Strong Scaling Test for AstroBEAR on Stampede]{
\includegraphics[width=.5 \textwidth]{TSFstrongOnStampede.eps}
%\label{fig:StrongTSF}
%}
%\subfigure[Weak Scaling Test for AstroBEAR on Stampede]{
\includegraphics[width=.5 \textwidth]{CFweakOnStampede_f.eps}
\label{fig:scalingPlot}
%}
%\captionof{figure}{(a) Strong scaling behavior of AstroBEAR with Triggered Star Formation module and $3$ levels of AMR on Stampede at TACC. Running time is plotted versus the number of cores in log scale. It has a slope $-0.84$ up to $1024$ cores which shows excellent scaling (slope $-1$ for "perfect scaling). The code performance drops when running on more cores (slope=$0.69$ up to $4096$ cores) of Stampede. (b) Weak scaling behavior of AstroBEAR with Colliding Flows module and fixed grid on Stampede. Runtime efficiency($1/t$ where $t$ is the running time) is plotted versus the number of cores in log scale. The work load on each core is kept same ( each has $64^3$ cells) for the weak scaling test. The communication efficiency of AstroBEAR drops only $15\%$ running on $1024$ cores and $27\%$ on $4096$ cores. }
\caption{Top: Strong scaling behavior of AstroBEAR with Triggered Star Formation module and $3$ levels of AMR on Stampede at TACC. Running time is plotted versus the number of cores in log scale. It has a slope $-0.84$ up to $1024$ cores which shows excellent scaling (slope $-1$ for "perfect scaling). The code performance drops when running on more cores (slope=$0.69$ up to $4096$ cores) of Stampede. Bottom: Weak scaling behavior of AstroBEAR with Colliding Flows module and fixed grid on Stampede. Runtime efficiency($1/t$ where $t$ is the running time) is plotted versus the number of cores in log scale. The work load on each core is kept same ( each has $64^3$ cells) for the weak scaling test. The communication efficiency of AstroBEAR drops only $15\%$ running on $1024$ cores and $27\%$ on $4096$ cores. }

\end{figure}

\section{Resource Request} 
{\color{green} Tables and Texts are added by Baowei on 4/10. Text and Numbers need to be revisited!!! }

For our colliding flows project, we propose to run 5 levels of AMR (instead of 4 levels as in our last proposal) and smaller base dimensions (comparing with our last proposal). This will increase the filling fractions and increase the effciency of using computing resources. Our estimate CPU-hours for this project is based on the strong scaling tests we've done which has the same resolution but running for only 0.1\% of one frame (we are planning to obtain 200 frames of the data). For our scaling test on 1024 cores, it takes about $144 seconds$ or $41 CPU-hour$ to run 0.1\% of one frame. So the total CPU-hours for one production run is about $820,000$ CPU-hours on Stampede. Since there's no I/Os when doing scaling test and it takes about $20~150$ seconds to save one frame of data (in format of chombo files, depending on the size of data). So we expect $\approx 1000$ CPU-hours to save the total 200 frames on 1024 cores for each run. So the total CPU-hours for each run is about 821,000 on Stampede (See Table(\ref{tab:SUs}).

When doing stron scaling tests for our triggered star formation project, we start from a point where the stars are just formed to get a better estimate. And we use the same base dimensions but lower (3 or 2) AMR levels (due to the availability of starting point at high levels). It took about 361 seconds to get 20\% of one frame from the starting points on 1024 cores which is about 103 CPU hours. And we estmate the running time will be 8 times longer when running with 4 levels of AMR. The current data size for one frame of 3-level-AMR is about 3GB and we estimate for our production runs the data size of each frame is about 6-12 GB. Depending on this information we estimate the total CPU-hours we need for each run of our triggered star formation project is 413,000 on Stampede (See Table(\ref{tab:SUs}).


In total we require 4.0 million CPU-hours on Stampede and 2.1 million CPU-hours on Gordon and 2.0 million CPU-hours on SuperMIC, about 98\% of which will be used for production runs and 2\% for testing runs and continue development of our code. We can use anywhere from 500-5000 cores for a typical production run, depending on queue limitations.

{\color{red} The following 2 paragraphs are from the old proposal !!!}
-------
Using AstroBEAR, we have found very interesting new results related to star formation - in particular studies of feedback (Quillen et al 2005, Cunningham et al 2009, Carroll et al 2009, 2010) as well as molecular cloud formation (Carroll et al 2013). We plan to carry out a series of three-dimensional numerical simulations of colliding flows with different shear angles, magnetic field strengths, and magnetic field orientations to study the star-forming properties of a cluster. In particular this will include colliding flows with $\beta=\infty$ and shear angles $0^{\circ}$, $30^{\circ}$ and $60^{\circ}$ as well as colliding flows with $\beta=1$ and $\beta=10$ and shear angles of $0^{\circ}$, and $60^{\circ}$, as well as colliding flows with $\beta=1$, a shear angle of $0^{\circ}$,  and magnetic field orientation angles of $0^{\circ}$ and $60^{\circ}$.  See Table \ref{tab:flows} This will allow us to address the interplay between magnetic field strength, orientation, and shear of the colliding flows with respect to the resulting properties of the molecular clouds as well as the cores that form within them.  
AstroBEAR is presently performing well with multiple 6+ levels of AMR refinement. The tractability of a given run then becomes more a question of the number of needed cell updates, which is mainly determined by the fraction of the volume where mesh refinement is employed. And the volume filling fraction depends on the specific problem/simulation. The computing resources we require are based on our previous runs on XSEDE machines. For example, Our hydro calculation with $0^{\circ}$ shear and with resolution $128^3 + 4$ AMR took about 6,000 SU's for 1 frame. In Tables \ref{tab:flows} we summary the computing resources we require. In total we require 7.1 million CPU-hours, 99.3\% of which will be used for production runs and 0.7\% for testing runs and continue development of our code. We can use anywhere from 500-5000 cores for a typical production run, depending on queue limitations.
-------

{\color{green} The following section is added by Baowei on 04/10/14}
\section{Introduction to Local Resources and Research Team}
We have a variety of computational platforms available locally for our research.  The Center of Integrated Research Computing(CIRC) at the University of Rochester hosts the Blue Gene/Q system called BlueStreak. BlueStreak consists of 1,024 nodes, 16,384 CPU cores, 16 TB of RAM, and 400 TB of storage. Each node consists of a 16-core A2 processor with 32 MB of cache and access to 16 GB of RAM. By reservation our group can have access to the whole rack(Normal reservations are for less than half of the rack, 24 hours per reservation). CIRC also hosts another Linux cluster called BlueHive which is expected to be in production in June of 2014. The new BlueHive system consists of approximately 200 nodes of IBM’s iDataPlex architecture with a high-speed, low-latency, InfiniBand interconnect. Each node houses 2 x 12-core Intel “Ivy Bridge” processors, with a range in memory from 64 GB to 256 GB. 30 of the 64 GB nodes have 2 dedicated NVIDIA K20X (“Kepler”) GPU (graphics processing unit) cards and 8 of the 64 GB nodes have 2 dedicated Intel Phi 5110P accelerator cards. Our group has access to 170 cores of the BlueHive system and 96 cores of them are owned by and dedicated to our group. Our group also has a dedicated 32 processor equivalent (PE) cluster built for developing and debugging AstroBEAR code. 

{\color{red} This part needs more details}
Our research group which currently has one faculty member, two computational science staffs, two postDocs, four graduate students and two undergraduate students has a very strong backgroup of computaional astrophysics......


\subsection{I/O Requirements, Analysis, and Storage}
Based on Table (\ref{tab:SUs}),  we expect to save 200 frames of data with size 5-20GB for each frame (the size of the each frame depends how many zones trigger the highest level of AMR) for each of the six runs of our colliding flows simulation with 5 levels of AMR. So the total data size for our colliding flows project is about 6 TB - 24 TB. We are planning to run these simulations on Stampede and we expect to need $\sim$24 TB of storage on Ranch of TACC. For each of of the four runs of our triggered-star-formation simulations, we expect to save 100 frams of data with size about 6GB, So the total data size for our triggered-star-formation project is about 2.4TB. These simulations will be done on Gordon. So we requestion $\sim$ 3 TB on Gordon.  Most of the analysis will be done using parallelized tools built into the AstroBEAR package that can be performed at run time or in post processing.

\subsection{Why choosing to run on these machines?}
{\color{green} this paragraph is added by Baowei on 04/10/14. Need revisit !!!}
We've been using Kraken and Stampede (before Ranger) for the past two very productive years. With Kraken decommissioned this April, we have to shift most our production runs to Stampede while seeking for opportunities on other machines. AstroBEAR behaves very good on Stampede (see the scaling tests attached). Since SuperMIC and Gordon have similar archetectures to Stampede, we expect the code will scale well on these two machines. For our Colliding flows project, we propose to run with higher level AMRs than before. So it will requires more memory usage. Gordon will be a good machine for these runs since it has big (64GB) RAM per node. Also we are planning to optimize the code on the Xeon Phi coprocessors of our local BlueHive system. So we expect the code will take the advantage of the MIC architecture and run even better on Stampede and SuperMIC. 
\subsection{Financial Support}
{\color{red} waiting for updated information from Sondra. Will be updated by Baowei!!}
Financial support for this project will come from the Space Telescope Sci Institute grant HST -AR-12128.01-A entitled ``STSci - Hubble Telescope - The Reel Deal: Interpreting HST Multi-Epoch Movies of YSO JetsSpace'' (PI, A. Frank; 10/1/2010 - 9/30/2013) and the Department of Energy grant DE-SC0001063 entitled ``The dynamics of magnetized Astrophysical Jets through Pulsed Power HEDP lab Studies'' (PI, A. Frank; 8/15/2012 - 8/14/2015) and the National Science Foundation, NSF  AST-1109285 entitled ``From Central Engine to Bipolar Outflow: Binaries, MHD and the Evolution of Planetary Nebulae'' (PI, A. Frank; 9/1/2011 - 8/31/2014). DOE, Award no. R17081, entitled ``Rice - Clumpy Environments \& Interacting Shock Waves: Realistic Laboratory Analogs of Astrophysical Flows'', (PI: A. Frank, 2/22/2011 - 2/21/2014) and the Space Telescope Sci Institute grant HST-AR-12832.01-A entitled ``Hubble Telescope Cycle 20 - Climbing the Ladder of Start Formation Feedback``, (PI A. Frank, 11/1/2012 - 10/31/2015).

\section{Summary}
{\color{red} Text from old proposal, need to updated !!!}
With the advent of our efficient 3D AMR MHD code AstroBEAR and state of the art HPC facilities, we are on the doorstep of significant breakthroughs in understanding the physics of the feedback process in the clustered star formation.  With our previous start-up XSEDE allocation (TG-AST120029), we have already found and are in the process of publishing very important results. We are in full production mode with AstroBEAR, but the powerful numerical resources of the XSEDE that we are requesting are needed to make further progress on unraveling the mysteries of interacting binaries. 

If granted this allocation, we will perform a series of 3D Adaptive Mess Refinement parallel numerical simulations with higher resolution and longer time-scales than ever before to study the formation, structure and stability and observational implications of inter-acting stellar binaries. 
\begin{itemize}
\item Study of colliding flows with MHD 4,930,000 SU's on Stampede.
\item Study of triggered star formation with/without MHD 10,000,000 SUs on Stampede.
\item Testing and continue development of our code. 7,000 SU's on Stampede; 0.7\% of total request. 
\end{itemize}

To achieve these goals, we request SU's on the TACC Stampede. 
%\bibliographystyle{plain}
%\bibliography{refs}
{\color{red} bibliography need to be updated !!!}
\begin{thebibliography}{}
%
\bibitem[Audit \& Hennebelle(2005)]{Audit2005} Audit, E., \& Hennebelle, P. \ 2005, \aap, 433, 1
\bibitem[Bally, J. 2011a]{bally11a}
Bally, J., 2011, Computational Star Formation, 270, 247 
\bibitem[Bally, J., Cunningham, N.J., Moeckel, N., et al. 2011]{bally11b}
John Bally, Nathaniel J. Cunningham, Nickolas Moeckel, Michael G. Burton, Nathan Smith, Adam Frank, and Ake Nordlund, 2011, \apj, 727, 113
\bibitem[Reipurth, B. and Bally, J.2001]{bally01}
Bo Reipurth and John Bally, 2001, ARAA, 39, 403
\bibitem[Bally, J., Licht, D.,Smith, N., & Walawender, J., 2006]{bally06} 
Bally, J., Licht, D.,Smith, N., \& Walawender, J., 2006, AJ, 131, 473
\bibitem[Bally, J., Reipurth, B., and Davis, C., 2006]{bally06b}
Bally, J., Reipurth, B., and Davis, C., 2006, Protostars and Planets V, University of Arizona Press, Tucson, in press
\bibitem[Ballesteros-Paredes et al.(1999)]{Ballesteros-Paredes1999} Ballesteros-Paredes, J., Hartmann, L., \& V{\'a}zquez-Semadeni, E. \ 1999, \apj, 527, 285
\bibitem[Ballesteros-Paredes \& Hartmann(2007)]{Ballesteros-ParedesHartmann2007} Ballesteros-Paredes, J., \& Hartmann. L. \ 2007, Revista Mexicana de Astronomia y Astrofisica, 43, 123
\bibitem[Burkert \& Hartmann(2004)]{BurkertHartmann2004} Burkert, A., \& Hartmann, L. \ 2004, \apj, 616, 288
%\bibitem[Ballesteros-Parades, Hartmann & Vazquez-Seandeni, 1999]{ballesteros99}
% Ballesteros-Parades, Hartmann \& Vazquez-Seandeni, 1999, ApJ, 527, 285
\bibitem[Carroll et al, 2010]{carroll10}
Carroll, J.J., Frank, A., \& Blackman, E.G. 2010, \apj, 722, 145 
\bibitem[Carroll et al, 2013]{Carrolletal13}
Carroll, J.J., Frank, A., \& Heitsch, F. 2013,"The effects of Inhomogeneities within Colliding Flows on the Formation and Evolution of Molecular Clouds", in preparation
\bibitem[Carroll et al, 2009]{carroll09}
Carroll, J.J., Frank, A., Blackman, E.G., Cunningham, A.J., 
\& Quillen, A.C. 2009, \apj, 695, 1376 
\bibitem[Carroll et al, 2011]{carroll11}
Carroll, J. Shroyer, B; Frank, A; Ding, C eprint arXiv:1112.1710. 2011, submitted to JCP
\bibitem[Cunningham et al, 2009]{cunningham09}
 Cunningham, A.J., Frank, A., Carroll, J., Blackman, E.G.,  \& Quillen, A.C. 2009, \apj, 692, 816 
\bibitem[Cunningham et al, 2009b]{cunningham09b}
Cunningham, A. Frank, A., Varniere, P., Mitran, S., \& Jones, T.W. 2009, \apjs, 182, 51 
\bibitem[Elmegreen et al, 2000]{elmegreen00}
Elmegreen B., G.,2000, \apj, 539, 342
\bibitem[Evans et al.(2009)]{Evansetal2009} Evans, II, N.~J., Dunham, M.~M., J{\o}rgensen, J.~K. et al. \ 2009, \apjs, 181, 321
 \bibitem[Hartmann et al.(2001)]{Hartmann2001} Hartmann, L., Ballesteros-Paredes, J., \& Bergin, E.~A. \ 2001, \apj, 562, 852
 \bibitem[Hartmann \& Burkert(2007)]{HartmannBurkert2007} Hartmann, L., \& Burkert, A. \ 2007, \apj, 654, 988
 \bibitem[Heitsch et al, 2011]{heitsch11}
Heitsch, F., Naab, T., \& Walch, S. 2011, mnras, 415, 271 
 \bibitem[Heitsch et al.(2008a)]{Heitsch2008a} Heitsch, F., Hartmann, L.~W., Slyz, A.~D., Devriendt, J.~E.~G., \& Burkert, A. \ 2008, \apj, 674, 316
 \bibitem[Heitsch et al.(2008b)]{Heitsch2008b} Heitsch, F., Hartmann, L.~W., \& Burkert, A. \ 2008, \apj, 683, 786
 \bibitem[Heitsch et al.(2009)]{Heitsch2009} Heitsch, F., Ballesteros-Paredes, J., \& Hartmann, L.~W. \ 2009, \apj, 704, 1735
 \bibitem[Hennebelle \& Audit(2007)]{Hennebelle2007} Hennebelle, P., \& Audit, E. \ 2007, \aap, 465, 431
 \bibitem[Huarte-Espinosa \& Frank(2012a)]{tenerife}
Huarte-Espinosa, M., \& Frank, A., 2012, arXiv:1201.4322
 \bibitem[Hueckstaedt(2003)]{Hueckstaedt2003} Hueckstaedt, R.~M. \ 2003, New Astronomy, 8, 295
 \bibitem[Klein et al, 2006]{klein06}
Klein, R., Inutsuka, S.,  Padoan, P., and Tomisaka, T ,  2006, Protostars and Planets V, University of Arizona Press, Tucson
 \bibitem[Krumholtz et al, 2005]{krumholtz05}
Krumholtz, M., McKee, K., \&  Klein, R., 2005, Nature, 438, 333
 \bibitem[Knee et al, 2000]{knee00}
 Knee, L.B.G., \& Sandell, G., 2000, ,aap, 361, 671
 \bibitem[Nakamura et al, 2011]{nakamura11}
 Nakamura, F., \& Li, Z.-Y., 2011, \apj, 740, 36 
 \bibitem[Nakamura et al, 2007]{nakamura07}
 Nakamura, F., \& Li, Z.-Y., 2007, \apj, 662, 395
 \bibitem[Matzner et al, 2002]{matzner02}
Matzner, C.D., 2002, ,\apj, 566, 302
 \bibitem[McKee \& Ostriker(2007)]{McKeeOstriker} McKee, C.~F., \& Ostriker, E.~C.\ 2007, ARA\&A, 45, 565
 \bibitem[Moriarty-Schieven et al, 2006]{moriarty06}
 Moriarty-Schieven, G.H., Johnstone, D., Bally, J., \& Jenness, T. 2006, \apj, 645, 357
 \bibitem[Norman et al, 1980]{norman80}
 Norman, C., \& Silk, J., 1980, \apj, 238, 158  
 \bibitem[Pringle et al.(2001)]{Pringle2001} Pringle, J.~E., Allen, R.~J., \& Lubow, S.~H. \ 2001, MNRAS, 327, 663
 %\bibitem[Reipurth et al, 2010]{reipurth10}
% Reipurth, B., Mikkola, 
% S., Connelley, M., \& Valtonen, M. 2010, ApJl, 725, L56 
 \bibitem[Quillen et al, 2005]{quillen05}
 Quillen, A.C., Thorndike, S.L.,Cunningham, A., Frank, A., Gutermuth, R.A., Blackman, E.G., Pipher, J.L., ,\& Ridge, N., 2005, ,\apj, 632, 941 
\bibitem[Smith et al, 2010a]{smith10a} 
Smith, N., Povich, M.S., Whitney, B.A., et al. 2010, mnras, 406, 952 
\bibitem[Smith et al, 2010b]{smith10b}
Smith, N., Bally, J., \& Walborn, N.R. 2010, mnras, 405, 1153
\bibitem[V{\'a}zquez-Semadeni et al.(1995)]{Vazquez-Semadeni1995} V{\'a}zquez-Semadeni, E., Passot, T., \& Pouquet, A. \ 1995, \apj, 441, 702
\bibitem[V{\'a}zquez-Semadeni et al.(2006)]{Vazquez-Semadeni2006} V{\'a}zquez-Semadeni, E., Ryu, D., Passot, T., Gonz\'alez, R.~F., \& Gazol, A.\ 2006 ApJ 643, 245
\bibitem[V{\'a}zquez-Semadeni et al.(2007)]{Vazquez-Semadeni2007} V{\'a}zquez-Semadeni, E., G{\'o}mez, G.~.C., Jappsen, A.~K., Ballesteros-Paredes, J., Gonz{\'a}lez, R.~F., \& Klessen, R.~S. \ 2007, \apj, 657, 870 
\bibitem[V{\'a}zquez-Semadeni et al.(2010)]{Vazquez-Semadeni2010} V{\'a}zquez-Semadeni, E., Col{\'{\i}}n, P., G{\'o}mez, G.~C., Ballesteros-Paredes, J., \& Watson, A.~W. \ 2010, \apj, 715, 1302
\bibitem[Warin et al, 1996]{warin96} 
Warin, S., Castets, A., Langer, W. D., Wilson, R. W., \& Pagani, L. 1996, A\&A, 306,  935
\bibitem[Wang et al, 2009]{wang09}
Wang, P., Li, Z.-Y., Abel, T., \& Nakamura, F. 2010, \apj, 709, 27 
\end{thebibliography}

\begin{appendix}
%\ctable[caption = Expected CPU-hours for Colliding Flow Simulations,
%        label = tab:flows
%       ]
%       {cccccc}
%       {}
%       {
%       \FL
%       Shear& $\beta$ & Field & Resolution & Frames & Expected\\
%       Angle&  & Orientation & & & SU's\\
%\ML
% $0^{\circ}$  & $\infty$  &  --  & $128^{3}+4$  & 150 & Completed \\
%\NN
% $30^{\circ}$  & $\infty$  & --  & $128^{3}+4$  & 150 &900,000 \\
%\NN
% $60^{\circ}$  & $\infty$  & --  & $128^{3}+4$ & 150 & 900,000 \\
%\NN
% $0^{\circ}$  & $1$  & $0^{\circ} $ & $128^{3}+4$ & 150 & Completed \\
%\NN
% $60^{\circ}$  & $1$  & $0^{\circ} $ & $128^{3}+4$ & 150 &1,050,000 \\
%\NN
% $0^{\circ}$  & $1$  & $30^{\circ} $ & $128^{3}+4$  & 150 &1,050,000 \\
%\NN
% $0^{\circ}$  & $1$  & $60^{\circ} $ & $128^{3}+4$  & 150  &1,050,000 \\
%\NN
% $0^{\circ}$  & $10$  & $0^{\circ} $ & $128^{3}+4$ & 150 &1,050,000 \\
%\NN
% $60^{\circ}$  & $10$  & $0^{\circ} $ & $128^{3}+4$ & 150 &1,050,000 \\
%\ML
% Total              &      &                &                    &   & 7,050,000\\
%\LL
% }
%
\ctable[caption = Estimate CPU-hours needed for each run (based on 1024 cores of Stampede,
        label = tab:SUs
       ]
       {ccc}
       {}
       {
       \FL
                    Project & Colliding Flows  & Tiggered Star Formation\\
\ML
 AMR levels for (Scaling VS Production) &  5 VS 5 & 3/2 VS 4 \\
\NN
 Num of frames (Scaling VS Production)  & 0.1\% VS 200 &20\% VS 100 \\
\NN
 WallTime/CPU-hour for Strong Scaling(1024 cores)  & 144 Secs/41 & 361 Secs/103 \\
\NN
 Data size of each frame  & 5-20 GB & 3GB (6-12GB with 4 AMR) \\
\NN
 Time needed to Save one frame  & ~ 20-150 Seconds  & ~22-120 Seconds \\
\NN
 Estimate CPU-hours for computing  & 820,000 & 412,000 \\
\NN
 Estimate CPU-hours for I/Os  &1,000  & ~800  \\
\ML
 Total Estimate CPU-hours & ~821,000 & ~413,000 \\
\LL
 }

\ctable[caption = Expected CPU-hours for Continuing Colliding Flows 3D Simulations,
        label = tab:CF
       ]
       {cccc}
       {}
       {
       \FL
                    Shearing Angle($\theta$) & Resolution  & Magnetic Field ($\beta$) & Expected SUs on Stampede\\
\ML
 0       & $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 15  &  $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 30  &  $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 0  &  $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 15  &  $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 20  &  $ 40^{3} $ + 5 AMR & 1 & 821,000\\
\NN
 30  &  $ 40^{3} $ + 5 AMR & 1 &821,000\\
\ML
Total &                  &  4,926,000
\LL
 }

\ctable[caption = Expected CPU-hours for 3D Triggered Star Formation Simulations,
        label = tab:TSF
       ]
       {cccc}
       {}
       {
       \FL
                    Hydro/MHD   & Resolution & Rotaion & Expected SUs on Stampede\\
\ML
 Hydro       & $320\times912\times192 $ + 4 AMR & Y &  413,000\\
\NN
 Hydro       & $320\times912\times192 $ + 4 AMR & Y &  413,000\\
\NN
 MHD $\beta=1$,Toroidal & $320\times912\times192 $ + 4 AMR & Y  &  413,000\\
\NN
 MHD $\beta=1$,Poloidal &  $320\times912\times192 $ + 4 AMR & Y & 413,000\\
\ML
Total &                         & 1,652,000
\LL
 }
\end{appendix}
\end{document}