NCSAstore
I wrote this script to deal with the scratch purge policies at NCSA. They have so many users that scratch space (where most people perform their runs) is purged of files older than a few days. Users are supposed to use the mass storage system, UniTree, for permanent storage of their output. The sample jobscript provided shows how to tack an upload to UniTree at the end of a job. However, it only uses one processor to do this, which means that all the others you've allocated (and written up a proposal to obtain) sit idle while the transfer happens.
Apart from obvious concerns about efficient use of the allocated resources, I ran into a further problem.
My code was producing some very large output files, and I wasn't getting particularly fast transfer times to UniTree.
Often, my jobs were being killed on their wall clock limits before they could save their output.
During my consultation with help desk, it emerged that each node had its own (fairly chunky) pipe to UniTree.
It was worth using the available nodes to do all the transfers at once.
Enter NCSAstore.pl.
NCSAstore.pl
Usage
NCSAstore is a PERL script.
Once you have downloaded it, you might need to tinker with the first line, so that it can find PERL on your system.
I've used something which usually works, but lots of things can go bump in the OS.
Basic usage is:
- destDir
- This is the destination directory on UniTree where you want the files to be stored
- nodeList
- This is a text file, containing a list of machines (one per line) to use to perform the parallel storage
- fileList
- This is a text file containing the list of files (one per line) to be stored
I wrote the script on the assumption that all the output files will be in one directory, and that they were all to be copied to the same directory on UniTree (a simple mirroring). If you want to copy directory trees, then this isn't going to help much, except as a starting point. Anyway, the script goes through each entry in fileList, assigns it to the next machine from nodeList, forks a copy of itself to instruct that machine to perform the copy, and then goes onto the next entry in fileList. Once all the machines are busy, it waits for all the child processes to complete before going through them all again. And again. And again, until all the entries in fileList have been stored.
So much for the theory. How about practical use? Well, my job scripts end like this:
# Designate storage directory on MSS
set STORAGE_DIR="Runs/Flash25/bhl/exe01"
# Make the file list
\ls -1 --hide-control-chars *hdf* *log flash.dat > files.txt
# Use my PERL script
$HOME/Flash-25/exe/NCSAstore.pl $STORAGE_DIR $PBS_NODEFILE files.txt
This is a csh script, so some details may change if your preferences are otherwise. Note that I use $PBS_NODEFILE for nodeList, since it's provided for free by the batch system. Also, \ls -1 --hide-control-chars makes absolutely sure that files.txt is formatted as it should be for fileList.
If you need to tinker with this script in order to get it to work with other computer systems, you are most likely to need to adjust $msscommand in the script. Additionally, $sshcommand might require some adjusting.
Disclaimer
Use at your own risk - I think that everything works, but I have been wrong in the past. In particular, this script does tend to assume that things work, which might not be a brilliant assumption always (although if it isn't, it's generally a problem for the help desk).
