Linux-Fu

Got something that helps you use Linux? Know of a neat trick, or just figure something out? Let everyone know about it here.

Command Line Reference

​Click here for a more complete listing of Linux commands, grouped by purpose.

Individually Tarballing Files

It is often useful to tarball files before transferring them between compute clusters and local storage. Large files should be rolled into separate tarballs to improve transfer efficiency.

Examples

To tarball chombo*hdf in a problem directory in bash, use a FOR loop:

user:~> for i in $(ls out/ch*hdf);   do tar -czvf $i.tar.gz $i [&& rm $i];   done

The bracketed rm $i command deletes the original chombo file, saving space.

You can get more creative with this. Say you've tar'd files 1-10, and now want to do 20-30. Let's also say that what you're really interested in is the later files, so you'd like to tar in reverse order. The following uses seq to generate a list of numbers in reverse order, which are then converted to 5-digit integers via printf and export:

user:~> for i in $(seq 30 -1 20); do 
  export num=`printf %5.5i $i`; 
  tar -czvf chombo$num.hdf.tar.gz out/chombo$num.hdf [&& rm out/chombo$num.hdf]; 
  done

Note the backticks on the export statement. The above can all be given on one line but is broken up here for clarity. Of course, if you want them to tar in normal order, you can simplify the above, as

user:~> for i in {20..30}; do export ...

With passwordless SSH, you can add a quiet scp statement for each tarball, scp chombo$num.hdf.tar.gz user@host:location &>/dev/null.

Finally, here's a more complicated statement related to tar'ing Brick-of-value *.dat files:

user:~> for i in {0..35}; do 
  export prefix="W_`printf %3.3i $i`" ;
  for j in $(ls $prefix*.dat); do 
    tar -czvf $j.tar.gz $j && rm -v $j;
  done;
done

Bash Shell Configuration: .bashrc and .bash_profile

Whenever you launch a bash shell via terminal, the shell environment is configured by the .bash_profile and .bashrc files in your home directory. The two files theoretically fulfill different roles, but the functionality they provide is very similar.

• The .bash_profile shell is executed when the you are logging in, be it through SSH, SFTP, or some other means. Basically, any launch that requires a username and password will execute the options .bash_profile.
• The .bashrc file, in contrast, is automatically executed when a non-login interactive shell is launched. For instance, if you are logged directly into a Linux machine and open a terminal window on the desktop, then .bashrc will be used instead of .bash_profile.

In practice, it's better to keep all of your environment settings in one of the two files. Otherwise, you'll have to change two files in order to change your shell environment. If for instance, a library path or module was changed in .bashrc and the change wasn't propagated to .bash_profile, then the new option might be unavailable for remote users (who log into the system, and therefore trigger .bash_profile).

For this reason, we usually put all of our environment configuration command in .bashrc and just add some lines to .bash_profile that invoke .bashrc:

if [ -f ~/.bashrc ]; then
        source ~/.bashrc
fi

Aside from this, .bash_profile is best kept relatively empty. This ensures that .bash_profile doesn't contain any settings that might override the ones in .bashrc.

Command Aliasing

In your home directory, .bash_profile executes when you open a terminal that requires a login (such as SSH), whereas .bashrc opens for a "non-login interactive shell", such as you might open in X with GNOME or KDE. In practice, .bash_profile will usually contain a line invoking .bashrc, so there's no effective difference.

There are several tricks with .bashrc to make your life easier. The first is the alias command, which maps complex shell-executable expressions to simpler commands using the form:

alias <command>="<bash shell expression>"

Examples

To always enable X11 forwarding in SSH:

alias ssh="ssh -Y"

To specify the build of VisIt in /opt/visit/bin to execute on the command visit:

alias visit="/opt/visit/bin/visit"

To apply .bashrc changes without logging out:

source ~/.bashrc

Exporting Environment Variables

Another useful trick in .bashrc is environment variable export. By including lines of the form export VARIABLE_NAME=<variable_value>, we make the variable $VARIABLE_NAME accessible within the command-line environment.

This is especially useful when applied to the $PATH and $LD_LIBRARY_PATH variables. These pre-existing environment variables contain the paths Linux searches to look for executables and shared library objects, respectively.

Examples

export PATH=$PATH:/usr/local/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hdf5/lib

Note the use of $PATH and $LD_LIBRARY_PATH in these variables. This concatenates the new path to any existing list of paths in the variable. Linux searches these paths in order, so if you want one path to appear before the rest you should use export <new_path>:$PATH.

Execute source ~/.bashrc after editing .bashrc to implement changes.

Persistent SCP connections

See the Persistent SCP page.

Regular Expressions in emacs

A regular expression (regex or regexp) is a special text string for describing a search pattern. The emacs editor supports regular expressions for finding and manipulating text.

Examples

I have several functions of the form C_name1(a,b,c), C_name2(a,b,c), etc., to rename with an extra underscore, i.e., C_name1_(a,b,c).

In emacs:

(M-X) replace-regexp
(1st prompt) \(C_\w*\)(
(2nd prompt) \1_(

In the first prompt (the to-be-replaced string), you can specify substrings using the \( and \) characters to group them. So I've wrapped the function name in such a group. The \w character will find any word character (no whitespace), and the * finds any number of them. It stops when it reaches the (, which is the next non-wildcard character in the to-be-replaced string.

Emacs allows you to have multiple groups in the to-be-replaced string, which you can reference in your replacement string. They are referenced left-to-right by \1, \2, etc. So here, I indicate that in the new string I want the first group (e.g. "C_name1") to come first, followed by a _(.

Conversely, if you want to pre/append an entire string (say, change all foot's to football's), you could use the \& character which represents the entire to-be-replaced string.

Monitoring the contents of a directory with 'watch'

Occasionally, a user wants to be able to watch a directory so that they can see when changes are made. An example is monitoring a problem directory on bg/p, in order to notice when a submitted job starts running since even if it says it starts immediately, output is delayed somewhat.

Examples

This can be done with the watch command, e.g. in your .bashrc:

alias wa='watch -d -n 1 ls -lht 2>/dev/null'

This will reexamine the current directory every second ("-n 1"), highlight changes ("-d"), and quietly ignore errors ("2>/dev/null"). Thus it is easy for the user to see when, for example, their problem *.out file is created and starts getting written to. Conversely, they know immediately when a file has quit and dumped core files, saving them the task of polling the directory manually or checking their email for a job-quit email message from the job scheduler.

Pushd/Popd

The pushd and popd set of commands allows Linux users to store directories on a stack and easily navigate between them.

Examples

Say you're debugging a problem module and you find that you're constantly switching between your problem directory (~/myprob) and the source directory (~/mycode/source). You can quickly bounce back and forth between the two with pushd and popd. pushd places your current directory and your destination directory onto a directory stack; you can subsequently alternate between the two by typing pushd without any arguments:

user:~/myprob> pushd ~/mycode/ user:~/mycode> cd source user:~/mycode/source> pushd user:~/myprob> pushd user:~/mycode/source> wow!

popd removes the current directory from the directory stack and puts you into the other directory; in general it would only be used to clear the stack.

pushd sets an environment variable $OLDPWD. This lets you greatly shorten the command to go back and forth from compilation and running the code. For instance, say you were editing the code in one terminal, and all you're doing in a second terminal is recompiling and running the code. You can do this all on one line like the following:

user:~/myprob> pushd ~/mycode && make mpibear && cp mpibear $OLDPWD && popd && mpirun -n 2 ./mpibear

Note that the use of "&&" instead of ";" in between commands will make the sequence halt if one exits with an error code (e.g., if there's a problem with compilation).

FORTRAN Command Line and Integer/String Read/Writes

See the FortranCommandLine page.

The Modules package

See the Modules page.