System Information

Spur (spur.tacc.utexas.edu), a Sun Visualization Cluster, contains 128 compute cores, 1 TB aggregate memory and 32 GPUs. Spur shares the InfiniBand interconnect and Lustre Parallel file system of Ranger, TACC's Sun Constellation Linux Cluster. Thus, Spur acts not only as a powerful stand-alone visualization system: it also enables researchers to perform visualization tasks on Ranger-produced data without migrating to another file system and to integrate simulation and rendering tasks on a single network fabric.

Spur is an 8 node visualization cluster (+ login node), with each node containing 16 cores, at least 128 GB of RAM, and 4 NVIDIA Quadro FX 5600 GPUs. The individual nodes are configured as follows: 

• spur: login node (no graphics resources)
• visbig : Sun Fire X4600M2 server with
  • 8 dual-core AMD Opteron processors
  • 256 GB RAM
  • 2 NVIDIA QuadroPlex 1000 Model IV (2 FX 5600 each)
• vis1: Sun Fire X4400 server with
  • 4 quad-core AMD Opteron processors
  • 128 GB RAM
  • 2 NVIDIA QuadroPlex 1000 Model IV (2 FX 5600 each)
• vis2-7: Sun Fire X4400 server with
  • 4 quad-core AMD Opteron processors
  • 128 GB RAM
  • 1 NVIDIA QuadroPlex 2100 S4 (4 FX 5600)

User Environment

Please note that this guide is specific to visualization and data analysis applications on Spur.  For more general information on the environment for Ranger and Spur, see the Ranger User Guide.

Graphics applications on Spur are served through a remote VNC desktop. VNC desktops are launched by way of SGE script, and visualization applications are then run directly under VNC.

Visualization and data analysis (VDA) applications must be run on a vis node (vis*.ranger.tacc), allocated to you by SGE. No VDA applications should be run on the Spur login node (spur.tacc). VDA applications running on the login node may be terminated without notice, and repeated violations may result in your account being suspended. Please submit a consulting ticket at https://portal.tacc.utexas.edu/consulting/ if you have any questions regarding this policy.

To launch a VNC desktop, execute the following commands:

1. ssh to spur: ssh < username >@spur.tacc.utexas.edu
    
2. If this is your first time connecting to spur, you must run vncpasswd to create a password for your VNC servers. This should NOT be your login password! This mechanism only deters unauthorized connections; it is not fully secure, as only the first eight characters of the password are saved. All VNC connections are tunnelled through ssh for extra security, as described below.
    
3. Launch a vnc desktop via SGE:
   qsub /share/sge6.2/default/pe_scripts/job.vnc
   for instance, to specify a particular account and desktop size, use:
   qsub -A TG-MyAcct /share/sge6.2/default/pe_scripts/job.vnc -geometry 1440x900
   
   This script can be copied to your home directory and modified, particularly if you would like to add your account information or change the default runtime of your job (currently limited to 24 hours). You can also change job runtime using the qsub command-line option "-l h_rt=< hours:minutes:seconds >".  To request a specific node, use the command line option "-l h=< node >".  For example, to request visbig, use "-l h=ivisbig". Note that you must put a leading 'i' before the node name.
   
   The default window manager is twm, a spartan window manager which reduces connection overhead. Gnome is available, if your connection speed is sufficient to support it. To use gnome, open the file ~/.vnc/xstartupand replace twm with gnome-session.
    
4. Once the job launches, connection info will be written to vncserver.out in your home directory. The very first time you run the VNC script, the vncserver file will not exist, so you can create it with the touch command.  You can then track when your connection information is written out using the tail -f command:
   touch ~/vncserver.out
   tail -f ~/vncserver.out 
    
5. The connection info will have a VNC port on spur for your session. However, for security, TACC requires that you tunnel your VNC session through ssh. You can set up a tunnel on a unix command line, or with a windows ssh client. You will need to select an arbitrary VNC port number on your local machine for the tunnel.
   
   From your local machine (not on the spur login), on unix command line, forward the port specified in your vncserver.out file to the matching port on spur, use the command:
   ssh -f -N -L 59xx:spur.tacc.utexas.edu:59xx < username >@spur.tacc.utexas.edu
   The '-f' instructs ssh to only forward ports, not to execute a remote command; the '-N' puts the ssh command into the background after connecting; and the '-L' forwards the port.   For example, to tunnel from 5951 on the local machine to 5951 on spur, use:
   ssh -f -N -L 5951:spur.tacc.utexas.edu:5951 < username >@spur.tacc.utexas.edu
   
   In a windows ssh client, find the menu where tunnels can be specified, and specify the local and remote ports as required, then launch the ssh connection to spur. For example, this image from PuTTY shows a tunnel from 5951 on the local machine to 5951 on spur:

    

   
   

6. Once the ssh tunnel has been established, use a VNC client to connect to the local port you created, which will then be tunnelled to your VNC server on spur. Connect to localhost:59xx, where 59xx is the local port you used for your tunnel. In the examples above, we would connect the VNC client to localhost::5951. Some VNC clients accept localhost:51.
   
   If you do not have a VNC client, these are available for Windows/Linux:
   TightVNC: http://www.tightvnc.com/
   TurboVNC: http://www.virtualgl.org/About/TurboVNC
   UltraVNC: http://www.uvnc.com/
   For Mac, we recommend:
   Chicken of the VNC:  http://sourceforge.net/projects/cotvnc/
   
   
7. After connecting your VNC client to your VNC server on spur, you may use visualization applications directly on the remote desktop without launching other SGE jobs. Any application that uses OpenGL library calls must be launched using vglrun , as described in the following sections.
    
8. When you are finished with your VNC session, you should kill your VNC server by typing exit in the black xterm window titled "*** Exit this window to kill your VNC server ***". Merely closing your VNC client will NOT kill your VNC server job on spur, and you will continue to be billed for time usage until the job ends. If you close your VNC client, you can reconnect to your VNC server at any time until the server job ends.

Usage

To access the vis stack on spur, you must load the vis module, if it is not already loaded: module load vis. Then list the available modules with module avail and the visualization applications will be listed under /opt/apps/vis/modulefiles. You must load an application's module, and possibly prerequisite modules, before launching the application.

To run applications that make use of the OpenGL library, you must precede the executable with vglrun. For example: vglrun visit . If you receive the error "Xlib: extension GLX missing on display" when launching an application, you must precede the executable with vglrun.

Running Parallel VisIt on Spur

After connecting to a VNC server on spur, as described above, do the following:

1. VisIt was compiled under the intel v10 compiler and mvapich v1.0.1 MPI stack. These must be loaded prior to running VisIt. Also, the default module 'CTSSV4' is incompatible with the VisIt server and must be removed. From the default environment, execute the following:
   module delete mvapich mvapich2
   module delete CTSSV4
   module swap pgi intel
   module load mvapich/1.0.1
   
2. If the vis module is not yet loaded, you must load it: module load vis
3. Load the VisIt module: module load visit
   
4. Launch VisIt: vglrun visit
5. Configure a parallel run engine:

    

   • Open the host profile: < Ctrl-H > or Options -> Host Profiles
   • Click the button "New Profile"
   • Under the "Selected profile" tab:
   • Name the profile, e.g. "spur parallel"
   • Remote host name will be the current vis node: localhost
   • Host name aliases: vis*.ranger.tacc.utexas.edu
   • This will permit this profile to be used from any node on spur
   • Check the "Parallel computation engine" box
   • (this activates the "Parallel options" tab)
   • Under the "Parallel options" tab:
     • Check the "Parallel launch method" box, and select "poe"
     • Be careful! There are several 'qsub' options, and you may need to scroll down to find "poe"
     • Set the "Default number of processors" field to a value greater than one. The exact value is ignored, but it must be 2 or more to avoid automatic launch of the serial engine. The actual number of processors requested is controlled by the number of nodes you requested for your VNC session.
   • Under the "Advanced options" tab:
     • Check the box "Use VisIt script to set up parallel environment"
     • Check the box "Tunnel data connections through SSH"
   • Click the button "Apply"
   • Click the button "Dismiss"
   • Save your configuration! Select Options -> Save Settings

For more information on VisIt, see https://wci.llnl.gov/codes/visit/home.html

Running Parallel VisIt on Ranger

VisIt can be run in batch mode on Ranger using software rendering, usually through VisIt’s python scripting interface. You must also load Mesa, a software-rendering implementation of OpenGL, because Ranger compute nodes lack GPUs. Here is a sample job script that launches visit in command-line mode and runs a python script:

#!/bin/bash

#$ -V				      # Inherit the submission environment
#$ -cwd 			      # Start job in submission dir
#$ -N visit.job			      # Job name
#$ -j y				      # stderr and stdout into stdout
#$ -o $HOME/$JOB_NAME.o$JOB_ID	      # Name of the output file
#$ -pe 16way 16			      # Request 1 Ranger node
#$ -q normal                          # Queue name
#$ -A TG-MyTGAcct		      # Account
#$ -l h_rt=01:00:00		      # runtime (hh:mm:ss) - 1 hour

# configure environment for visit
module purge
module load TACC
module delete pgi mvapich CTSSV4
module load intel mvapich
module load vis mesa visit/1.10.0

# run visit
visit -cli -nowin –s myVisItScript.py

To open a parallel engine in your python script, use the following:

launchArguments = (“-par”, “-np”, “16”,  “-l”, “poe”)
OpenComputeEngine(“localhost”, launchArguments)

Just like when using VisIt on Spur, the number of parallel engines launched is controlled by the job script ‘-pe’ option, not the engine ‘-np’ argument, but the –np argument must be 2 or more so a parallel engine will be used.

A manual for VisIt’s python interface is at: https://wci.llnl.gov/codes/visit/manuals.html

Running Parallel ParaView on Spur

After connecting to a VNC server on spur, as described above, do the following:

1. ParaView was compiled under the intel v10 compiler and OpenMPI v1.3 MPI stack. These must be loaded prior to running ParaView. However, they are not loaded by default. You must load them manually:
   module delete mvapich mvapich2
   module swap pgi intel
   module load openmpi/1.3
   
2. If the vis module is not yet loaded, you must load it: module load vis
3. Load the ParaView module: module load paraview
   
4. Launch ParaView: vglrun paraview
5. Launch a parallel ParaView server:
   • In a separate xterm, load the modules as described above
   • After all modules have been loaded, launch the ParaView server. To launch with parallel rendering (which will open additional rendering windows) use:
     ibrun vglrun pvserver
     When these windows open, you can minimize them, but do not close them.
      
   • ibrun will configure the parallel environment and launch the ParaView server. When the server is ready, it will print:
     

     Listen on port: 11111
     Waiting for client...

   • Connect to the server from within the ParaView client:
     • Click the "Connect" button, or select File -> Connect
     • Click "Add Server"
     • Enter a "Name", e.g. "manual launch"
     • Click "Configure"
     • For "Startup Type", select "Manual"
     • Click "Save"
     • select the name of your server configuration, and click "Connect"
     • In the xterm where you launched ParaView server, you should see "Client connected."

For more information on ParaView, see http://www.paraview.org/

Running IDL on Spur

To run IDL interactively in a VNC session, connect to a VNC server on spur as described above, then do the following:

1. If the vis module is not yet loaded, you must load it: module load vis
2. Load the IDL module: module load idl
3. Launch IDL: idl
   or
   Launch the IDL virtual machine: idl -vm

If you are running IDL in scripted form, without interaction, simply submit an SGE job to the 'vis' queue that loads IDL and runs your script. The 'vis' queue only allocates to spur's vis nodes.

If you need to run IDL interactively outside of a VNC session, you will need an SGE job running in the vis queue so that you are allocated a vis node. The vncserver job is an easy way to do this, though submitting a script with 'sleep N' where N is the amount of time you will use the node would work just as well. The point is to allocate a vis node to you.

Once you have a node allocated, you can ssh to it (ssh -X vis[1-7,big]). However, these are not reachable from outside the Ranger/Spur firewall, so you must first ssh to a login node (spur, login3.ranger, login4.ranger).

To summarize: once you have a vis node allocated, you can ssh through a login node to the allocated vis node.

Once you're done, you can let the SGE job expire naturally (though you will be charged for the time used), or you can qdel it to terminate it immediately.

Running Amira on Spur

Amira runs only on node vis6 of spur. You must request this node explicitly in qsub, either in your script or on the command line using the argument "-l h=ivis6". Note the leading 'i'!

qsub -l h=ivis6 -A TG-MyAcct /share/sge/default/pe_scripts/job.vis

After connecting to a VNC server on spur, as described above, do the following:

1. If the vis module is not yet loaded, you must load it: module load vis
2. Load the Amira module: module load amira
3. Launch Amira: vglrun amira

Resources

If you do not see your question answered here, please submit a consulting ticket at https://portal.tacc.utexas.edu/consulting/

General Questions

• Why VNC?
  Must I use VNC to use Spur?
  Can I use X-forwarding with Spur?
  VNC is the preferred method for using graphics-based applications on Spur, as it is more efficient that X-forwarding.  Only keyboard, mouse, and screen updates are sent over the remote connection.  However, X-forwarding will work on Spur.  
  
  You may X-forward terminals, editor sessions, etc from the login node.  Visualization and data analysis apps may be X-forwarded as well, but they must be run on a vis node (vis*.ranger), allocated by SGE, not the login node (spur).  The vncserver job is an easy way to allocate a vis node, though submitting a job script with 'sleep N' where N is the amount of time you will use the node would work just as well. The point is to allocate a vis node to you.
  
  Once you have a node allocated, you can ssh to it (ssh -X vis[1-7,big]). However, these are not reachable from outside the Ranger/Spur firewall, so you must first ssh to a login node (spur, login3.ranger, login4.ranger).
  
  
  
• Must I ssh-tunnel my VNC connection?
  No, though this practice is not recommended.  VNC communications are not encrypted, so keystrokes can be logged.  Never type a password inside an unencrypted VNC session!  Tunneling the VNC connection with ssh encrypts the communication, with negligible effect to performance, thus removing this security vulnerability.
   
• My argument to qsub was not recognized
  My argument to vncserver was not recognized
  Arguments to qsub must be put before the script filename, arguments to vncserver must be put after the script filename. See the example at point #3 under "Logging into Spur", above.
   
• My script could not find the module for a visualization application
  ModuleCmd_Load.c(199):ERROR:105: Unable to locate a modulefile for '<vis app>'
  The 'vis' module, the parent module for all vis applications, is loaded by default on Spur, but not on Ranger.  If a vis job is launched from a Ranger login node, to the vis queue or any other queue, the vis module must be loaded explicitly.  Putting 'module load vis' in your job script will make sure the vis modules are available when launched from Ranger, and it will still work if the job is launched from Spur.
   
• My application fails with GL errors
  Xlib: extension "GLX" missing on display ":1.0"
  Your application uses the OpenGL library, and must be run with vglrun on a vis node: vglrun < app >
   
• "vglrun" command not found
  vglrun is installed only on the vis nodes of spur (vis*.ranger), not on the login node (spur). If your prompt contains "spur", you are on the login node. You must be in a VNC session on a vis node to use vglrun and OpenGL-based applications.
   
• How can I get more RAM for my application?
  My application cannot use all the RAM available
  My application hits a memory limit
  SGE sets per-process memory limits based on the ‘wayness’ of the SGE job, as specified in the ‘-pe’ argument of your job script.  The default script uses ‘-pe 16way 16’, which permits 16 parallel processes, each with a memory limit of 1/16th available RAM (8GB on vis1-7, 16GB on visbig).  Selecting a lower wayness will permit fewer parallel processes, each with a higher memory limit.  For example, using ‘-pe 4way 16’ permits 4 parallel processes, each with a memory limit of 1/4th available RAM (32GB on vis1-7, 64GB on visbig).  Note that the memory limit is enforced at the kernel level, while parallel process limit applies only to MPI environments created by the TACC ibrun script.  Multi-threaded programs not using ibrun are subject only to the memory limit.

Amira

• Amira cannot find a valid license
  Amira only runs on vis6. Make sure that you are running on vis6 (look at the prompt in your VNC session terminals, or see your vncserver.out file). You must explicitly request vis6 through qsub, as described in "Running Amira on Spur", above.