Introduction to iRODS at TACC

iRODS is a data grid/data management tool. It allows you to store data in a unified namespace using multiple storage resources, to replicate data so that copies exist on multiple systems, and to store checksums and arbitrary metadata with a file. The TACC iRODS configuration supports accessing iRODS through either the native iRODS tools such as the UNIX "i-Commands" or through WebDAV. iRODS is configured to store data on a 20TB "cache" filesystem for relatively short-term storage, on the 750TB "Corral" Lustre file system, and on any of the 3 Ranch archive file systems. Each of these storage systems is referred to as a "resource" within iRODS. The resources names are documented in the following table:

Use of the Corral resource may be subject to allocation constraints - please limit yourself to using the Ranch archive file systems for long-term storage unless you have an allocation on Corral.

Setup for command-line usage

On Ranger and Lonestar:

The directory /opt/apps/irods contains an example configuration file, irodsEnv.

1.  cp /opt/apps/irods/irodsEnv  ~/.irods/.irodsEnv

2.  edit ~/.irods/.irodsEnv  to include your username in the indicated locations

3.  Add /opt/apps/irods/bin to your path so you can access the binaries, or use "module load irods"

4.  Run the "iinit" command to initialize the system - this will ask for a password, which you only have to enter once per session

Basic Command-line Usage

Once you have configured the ~/.irods/.irodsEnv file, you can use the "i-commands" to access data in the system. The i-commands are generally the same as the standard Unix file management commands, but with an i prepended: ils, imkdir, icd, and so on. The "-R" switch is used to specify a target resource for commands that store data in the system. Other common switches include

-v for verbose output

-r for recursive operation

-h for detailed information on usage of any given command

The "ils -l" command can be used to see all the copies of files in the system. If a file has been replicated from corral to ranch3, for example, the file wil be listed twice, with each listing indicating the resource where the file or replica is stored, along with the replica number, which is used in commands like irm or iget, which can target a specific copy of a file.

Storing data into and retrieving data from the iRODS system

The “iput” command is used to store data into the system

iput /home//

If you used the default .irodsEnv file, this will store data in the largest Ranch file system,  /home3. To specify a resource target use the “-R” switch.

Including the “-K” switch will cause a checksum to be generated when the data is stored.  The user has the option of verifying this checksum when retrieving the data with the “iget” command.

The “iget” command is used to retrieve data from the system.

iget

Including the -K switch will trigger verification of the checksum the user may have generated when storing the data using the “iput” command.

Long-term storage of data from the cache

To replicate data stored in the cache file system into Ranch for long-term storage, use the "irepl" command as follows:

irepl -R ranch3

The command above will replicate the cache file into the Ranch resource target ranch3.

Deleting data from one or all resources

Use the “irm” command to delete data entirely or from a single resource. irm without options deletes all copies of a file. irm with the "-n #" switch deletes a specific replica. If you have stored data initially in the cache and then replicated it to Ranch, for example, replica 0 will be the copy stored on the cache, and the command:

irm -n 0 <path-to-file>

will remove the file from the iRODS cache resource, while leaving the archived copy intact. Use "ils -l" before deleting a replica to ensure that you have a copy in more that one resource, and that you are deleting the correct replica.

Synchronizing a local directory with iRODS

The irsync command can be used to synchronize a local directory with iRODS, similar to the rsync Unix command. It can be used to make an exact copy of a directory hierarchy on a local disk within iRODS, or retrieve an exact copy of a directory hierarchy already stored in iRODS. It may also be used to create an exact copy of a file or directory within iRODS. iRODS paths are identified with an i: prefix in the irsync command. For example, if you have created a directory within iRODS called /tacc/home/joeuser/myproject, and you wish to retrieve an exact copy of that directory on Ranger, run the command:

irsync -r i:/tacc/home/joeuser/myproject /path/to/joeusers/work/directory

After editing the files on Ranger, you can then synchronize the data back into iRODS using the command:

irsync -r /path/to/joeusers/work/directory i:/tacc/home/joeuser/myproject

If you are storing or retrieving data to Ranch with the -R ranch3 option, you should also use the -s switch - this will use the size rather than the checksum of the file to determine whether synchronization is necessary, thereby avoiding the need to retrieve all the files from tape to compute checksums. This will greatly improve the performance of synchronization with Ranch.

This is from the Lonestar User Guide; good source of general advice regarding optimization. - JBS

The most practical approach to enhance the performance of applications is to use use advanced compiler options, employ high performance libraries for common mathematical algorithms and scientific methods, and tune the code to take advantage of the architecture. Compiler options and libraries can provide a large benefit for a minimal amount of work. Always profile the entire application to ensure that the optimization efforts are focused on areas with the greatest return on the optimization efforts.

"Hot spots" and performance bottlenecks can be discovered with basic profiling tools like gprof. Observe the relative changes in performance among the routines when experimenting with compiler options. Sometimes it might be advantageous to break out routines and compile them separately with different options than those used for the rest of the package. Also, review routines for "hand-coded" math algorithms that can be replaced by standard (optimized) library routines. You should also be familiar with general code tuning methods and restructure statements and code blocks so that the compiler can take advantage of the microarchitecture.

Code should:

• be clear and comprehensible
• provide flexible compiler support
• should be portable

Avoid too many architecture-specific code constructs. Use language features and restructure code so that the compiler can discover how to optimize code for the architecture. That is, expose optimization when possible for the compiler, but don't rewrite the code specifically for the architecture.

For single processor optimization, the first step involves a moderate amount of hand-tuning. However, excessive amounts of hand-tuning can lead to loss of clarity and limits compiler flexibility. Clean up code to avoid restricting the compiler and to expose optimization opportunities. Let the compiler do most of the optimization for you.

Some best practices:

• Avoid excessive program modularization (i.e. too many functions/subroutines)
  • write routines that can be inlined
  • use macros and parameters whenever possible
• Minimize the use of pointers
• Avoid casts or type conversions, implicit or explicit
• Avoid branches, function calls, and I/O inside loops
  • structure loops to eliminate conditionals
  • move loops around a subroutine, into the subroutine

This usually takes care of the majority of changes necessary for moderate hand-tuning to obtain optimal code. After hand-tuning, the compiler options typically lead to the biggest improvement in performance. So, devote some time to understanding the meaning and significance of the recommended options. After these basic steps, use profiling to locate "hot spots" or performance bottlenecks. Make use of performance routines provided by vendor libraries, as opposed to writing your own version.

Remember the following principles when setting a password:

• Protect your password.
  • Don't  write  down your password - memorize it.  In particular, don't write it down and leave it anywhere, and don't place it in an unencrypted file!
  • Use unrelated passwords for systems controlled by different organizations.
  • Don't give or share your password, even to someone claiming to be from computer support or a vendor.
  • Don't let anyone watch you enter your password. Don't enter your password to a computer you don't trust.
  • Use the password for a limited time and change it periodically.
• Choose a hard-to-guess password.
  • The password setting interface will prevent you from choosing an insecure password. Even so, choose your password wisely.
  • Don't use something you'd find in a dictionary (in any language or jargon). Don't use a name (including that of a spouse, parent, child, pet, fantasy character, famous person, and location) or any variation of your personal or account name. Don't use accessible information about you (such as your phone number, license plate, or social security number) or your environment. Don't use a birthday or a simple pattern (such as backwards, followed by a digit, or preceded by a digit.
  • Instead, use a mixture of upper and lower case letters, as well as digits or punctuation. When choosing a new password, make sure it's unrelated to any previous password. Use long passwords. You might use a word pair with punctuation inserted, a phrase (an understandable sequence of words), or the first letter of each word in a phrase.
• To change your NCSA (teragrid) password, visit this website:

https://internal.ncsa.uiuc.edu/cgi-bin/CC/services/passwords/index.cgi

Follow the instructions on the page. To view when your NCSA password expires, you can refer to this page:

https://internal.ncsa.uiuc.edu/cgi-bin/CC/kerberos/expiration.pl

• To change your TACC password, use this URL:

https://portal.tacc.utexas.edu/

https://tas.tacc.utexas.edu/TASMigration/ChangeTASPassword.aspx

Debugging with DDT

DDT is a symbolic, parallel debugger that allows graphical debugging of MPI applications. DDT does support threads, but this requires additional manipulation of the “template” job script.

Code Preparation

Before running with any debugger, go to your application directory and re-compile your code with -g -O0. For example:

login3% mpif90 -g -O0 debug_code.f90

System Preparation

For Ranger, please follow these steps to set up and your environment:

1. Make sure that X11 forwarding is enabled when you ssh to Ranger. Use the -X option on the ssh command line if X11 forwarding is not enabled in your ssh client by default.
   

    

   To test, login to Ranger and check the display back to your workstation. If a clock appears on your screen when you execute "xclock", then X11 forwarding is working. Here is the general command line syntax to use with ssh, from a UNIX/Linux system:

   % ssh -X ranger.tacc.utexas.edu

    

2. DDT is installed only on the frontend nodes. While logged in, load the ddt module file as follows:

   login4% module load ddt

    

Configuring DDT

1. With the DDT module loaded in addition to all other modules needed to run your program (a.out in the example below), start the DDT debugger as follows:

   login4% ddt ./a.out

    

2. Click the "Run and Debug a Program" button in the "DDT – Welcome" window.

   The DDT Welcome window

    

3. This displays the "DDT-Run" window, where you specify the executable path, command-line arguments, and processor count. Once set, they remain from one session to the next.

   The DDT Run window

    

4. Select each of the "Change" buttons in this window, and adjust the job parameters, as follows:
   1. For the "Options" change button, set the MPI Implementation to either "mvapich 1" or "mvapich 2", depending on which mpi software stack you used to compile your program. Click OK.

      The DDT Options window

       

   2. For the "Queue Submission Parameters" window, fill in the following: Queue (development is the default queue, and it has fast turnaround, use the normal queue for more than 256 cores); Time (hh:mm:ss); and Project to charge. You must set the Project if you are assigned more than one. When you login, there is a list of the projects associated with your account. Click OK, and you will return to the "DDT-Run" window.

      The DDT Queue Submission Parameters window

       

5. Back in the "DDT-Run" window, set the number of cores you will need in the "Number of processes" box (if you are not using a multiple of 16, see instructions below).
6. Finally, click "Submit". A "DDT-Job Submitted" status box will appear.

Running DDT

When your job runs, the status box will disappear. Double click on line numbers to set break points, and then click on the play symbol (>) in the upper left corner to start the run.

The DDT Session window

The status of a queued job is updated in this window. When your job is launched by the SGE scheduler, the DDT GUI will fill up with a code listing, stack trace, local variables and a project file list.

Advanced Use

See the DDT User Guide for details on navigating around the GUI, managing the job, and viewing data.

If you need to run a hybrid code and/or use a number of cores that is not divisible by 16 (the number of cores on a node), please contact TACC support. To do that, you will need to create your own template, similar to the one in your ~/.ddt directory, config.ddt. Here are general instructions for creating a custom template:

Look in your .ddt directory and find the ddt configuration file called "config.ddt". In that file, find the section that looks like this:

The sge.qtf file is the job submission script config.ddt tells DDT to use. Either copy that script to your home directory so you can change it and modify config.ddt to use your modified script, or change config.ddt to use a different job submission script.