Skip Nav

Frequently Asked Questions

Table of Contents

1. Logging In

Q. How do I change my password?

A.Use the Kerberos kpasswd command on your local host or on HPC servers. The UNIX command, passwd, is not valid on systems using Kerberos for secure login. Windows users may also use the "Change Password" button when using KRB5/kinit.

Q. After entering my password and pressing my YubiKey button, I am still unable to get a ticket. What else can be wrong?

A.Several things could cause your password and YubiKey to stop working. These things include a YubiKey that is out of sync, system outages, your YubiKey not working, or multiple incorrect login attempts. The quickest way to resolve the problem is to contact CCAC so the card can be checked.

Q. I want to install the latest Kerberos software. Where do I get it?

A.The official DoD HPCMP Kerberos software can be downloaded from HPC Centers: Kerberos & Authentication.

Q. How do I register my CAC with pIE?

A.Perform the following steps to register you CAC with pIE:

  1. Go to https://ieapp.erdc.hpc.mil
  2. Click the "Accept Conditions and Continue" button on the US Government Notice and Consent Banner.
  3. Click the "OpenID Login" button, and complete the OpenID login process.
  4. Once logged in, click on the tab titled "Register CAC to Your Existing Account," and follow the instructions to register your CAC.

    It usually takes a day for your CAC information to be entered into the Kerberos KDC. You may or may not receive a notice after this has been done.

Q. How can I view my account usage and remaining allocation?

A.The command "show_usage" is available on all computational systems and reports allocations and usage. This output is displayed at login, but users can run the command: show_usage at any time to see the output.

Q. How do I login to an ARL DSRC machine using Kerberos on a PC?

A.Follow these steps:

  1. Run KRB5.EXE. The following dialog will appear:
    Kinit interface
  2. Enter your HPCMP userid in the Name field.
    Enter HPCMP.HPC.MIL (in all caps) in the Realm field.
    If using a CAC card and PKINIT, leave the Password field blank and press the <ENTER> key.
    If using a YubiKey, enter your HPCMP Kerberos password into the Password field and press the button on the YubiKey in USB port.
    After a few seconds, you will get a green Kerberos ticket that is valid for 10 hours.
  3. Open PuTTY and enter the information below:
    Putty interface showing how to save a session.

    Replace the username with your userid.
    After you have done this, name the connection (for instance, Pershing) under "Saved Sessions" and click save (red circle).

  4. Now, whenever you want to connect to that system, double-click on the system name, as shown below:
    Putty interface showing how to open a saved session.

Q. I get the following error when getting a ticket through my YubiKey: "kinit: Preauthentication failed while getting initial credentials". How do I solve this?

A.For example:

smith@blue> kinit smith@HPCMP.HPC.MIL
Password for smith@ARL.HPC.MIL:
Passcode:
kinit: Preauthentication failed while getting initial credentials
          

There are three possible causes:

  • A bad password.
  • A bad passcode. This could be due to a malfunctioning YubiKey.
  • You don't use the kdc_timesync = 1 and ccache_type = 4 in your Kerberos configuration, and the time on the machine has drifted. If this is the case, try checking your configuration file and adjusting your system clock.

Try to kinit a couple times to make sure you have entered the "correct" password and passcode. If you are still having the same problem, please send an email to help@ccac.hpc.mil.

Q. What do I do when I get: "Can't send request (send_to_kdc)"?

A. Your system can't get the correct address for the Kerberos at your principal realm. If you are a new Kerberos user, you need to check your Kerberos configuration file (krb5.ini or krb5.conf). Check the entries under [default_realm] and [domain_realm].

#Kerberos Services
klogin          543/tcp         # Kerberos authenticated rlogin
kshell          544/tcp   cmd   # and remote shell
eklogin        2105/tcp         # Kerberos encrypted rlogin
Kerberos         88/udp   kdc   # Kerberos authentication--udp
Kerberos         88/tcp   kdc   # Kerberos authentication--tcp
kerberos-sec    750/udp         # Kerberos authentication--udp
kerberos-sec    750/tcp         # Kerberos authentication--tcp
kerberos_master 751/udp         # Kerberos authentication
kerberos_master 751/tcp         # Kerberos authentication
kerberos_adm    752/tcp         # Kerberos 5 admin/changepw
passwd_server   752/udp         # Kerberos passwd server
kpop           1109/tcp         # Pop with Kerberos
kshell          544/tcp   cmd   # and remote shell
klogin          543/tcp         # Kerberos  authenticated rlogin
eklogin        2105/tcp         # Kerberos encrypted rlogin
kftp            765/tcp         # Kerberized ftp 
krb_prop        754/tcp         # Kerberos slave propagation

Q. Why am I getting the error message "clock skew too great" when requesting a Kerberos ticket using the Windows client?

A.This indicates that the clock on your computer has the wrong time.

"Clock skew" is the range of time allowed for a server to accept Kerberos authenticators from a client. In order for Kerberos authentication to work, your Windows client and the Kerberos server's time need to be within 5 minutes of each other. If they are too far off you will receive the "clock skew too great" error message and you will not be able to get a Kerberos ticket.

To resolve this issue you must manually set the clock on your system to the correct time.

To avoid this problem in the future there are several free time synchronization programs available to use under Windows. Here are a couple of easy to use programs for keeping the time current on your Windows system:

They both have install/uninstallers and require no knowledge of NTP (Network Time Protocol).

If you are running Windows XP, Vista, or Windows 7 you will need to have administrator privileges in order to set your time, either manually or using an automated program.

For Admins of XP, Vista, or Win7 boxes there is a Windows port of the Unix NTP code. This might be useful if your site already has NTP servers setup, and you want to sync your Domain controller to it (and have your clients sync to Domain controller using Microsoft utils): http://www.five-ten-sg.com/util/ntp4172.zip.

Q. What firewall settings should I have set locally to access the ARL DSRC?

A.If there is a Firewall between your machine using Kerberos and the ARL DSRC and you are unable to connect, provide the following information to your firewall administrator:

Different Kerberos clients sometimes contact different ports for the same services. Kerberos servers know how to respond to the various clients. Random client ports usually run from 1024 to 65536, but some ssh clients use priviledged ports 1023, 1022, 1021, ... for each successive simultaneous ssh connection.

A site should open all the ports listed below:

Service                 TCP/UDP         Server Port     Client Port
kinit/krb5.exe          tcp             88              random
kinit/krb5.exe          udp             88              88
kpasswd                 tcp             749             random
kdc-a.afrl.hpc.mil      tcp             749             random
kshell/rcp/rsh          tcp             544             random
kshell/rcp/rsh          tcp             1023,1022,...   random
encrypted rlogin -x     tcp             2105            random

A site should also open the ports for kftp. This uses the standard ftp ports:

Service                 TCP/UDP         Server Port     Client Port
Ssh                     tcp&udp         23              random
ftp data                tcp             random          random

The ARL DSRC hosts accept ssh from ssh clients that know how to use Kerberos credentials. Windows, Linux, and Mac versions are available on HPC Centers: Kerberos & Authentication. SSH should then be used to tunnel X11 sessions securely as well as for regular ssh connections.

Sites should also open this port:

Service                 TCP/UDP         Server Port     Client Port
ssh                     tcp             22              random

DoD networks may block standard X11 ports now or in the near future. These ports generally start at 6000 and work up for additional Xdisplays. One DoD suggestion is to block ports often used by X11 from 6000 tcp/udp to 6063 tcp/udp. This would have an adverse affect of most tcp/udp protocols. All tcp/udp protocols choose random ports from the range 1024-65536. When they get a failure, they often increment the port number by 1. So a valid process unrelated to X11 which happens to choose random port number 6000 could have to retry 64 times before getting an unblocked port number.

If the site permits X11, and X11 tunneled via SSH is not available, open the following ports (note, here the server is on the machine where the X11 display is running - at the user-site rather than at the ARL DSRC):

Service                 TCP/UDP         Server Port     Client Port
X11                     tcp             6000            random
X11                     tcp             6001            random
X11                     tcp             6002            random
X11                     tcp             6003            random

(Allow the most common X11 ports. Permit numbers greater than 6003 if firewall/filter logs show valid traffic getting blocked.)

These ports may be used by some Kerberos clients. Only open these if filter/firewall logs show that valid traffic to these ports is getting blocked:

Service                 TCP/UDP         Server Port     Client Port
kftp                    tcp             765             random
ftp-data                tcp             20              random
Example filter configuration for a Cisco router

These filters would be installed at a user site and applied to traffic coming inbound. Without comments, the configuration looks like this:

access-list 101 permit tcp any eq 88 any
access-list 101 permit udp any eq 88 any
access-list 101 permit tcp any eq 749 any established
access-list 101 permit tcp any eq 544 any established
access-list 101 permit tcp 140.31.64.0 0.0.63.255 range 1015 1023 any
access-list 101 permit tcp any eq 2105 any established
access-list 101 permit tcp 140.31.64.0 0.0.63.255 eq 23 any established
access-list 101 permit tcp 140.31.64.0 0.0.63.255 eq 21 any established
access-list 101 permit tcp 140.31.64.0 0.0.63.255 any established
access-list 101 permit tcp any eq 22 any established

Q. My login node response time is sluggish? How do I find a less busy node?

A.You can use the node_use command to determine the current least busy node available. The command format is:

node_use

It provides the memory usage and load average for all the login nodes. For example:

Pershing-login02> node_use 

Login Node Memory Status

Node Name           Total (Kb)   Used (Kb)   Free (Kb)   Pct. Free   Load Avg.
================    ==========  ==========  ==========  ==========  ==========
pershing-login01    65779756    52898100    12881656      19.58%        0.60
pershing-login02    65814048    64202936     1611112       2.45%        0.42
pershing-login03    65780416    64863984      916432       1.39%        2.14
pershing-login04    65779756    64064664     1715092       2.61%        1.49
pershing-login05    65779752    63791012     1988740       3.02%        0.98
pershing-login06    65779524    59188196     6591328      10.02%        2.19

Q. What local shell variables are automatically defined in my login environment?

A.The Baseline Configuration Team has determined a set of environment variables to be defined at all centers. Details can be found at http://centers.hpc.mil/consolidated/bc/policies.php?choice=environment

Q. How do I log into an ARL DSRC machine using my YubiKey?

A. Please review http://centers.hpc.mil/users/yubi.html.

2. Machine Configuration

Q. What is the configuration of your IBM iDataPlex (Pershing)?

A.A thorough configuration summary for Pershing is available in the System Configuration section of the Pershing User Guide.

3. Manuals

Q. Where can I find the Pershing User Guide?

A.On the ARL DSRC Web site at http://www.arl.hpc.mil/docs/pershingUserGuide.html

Q. Where can I find the Utility Server User Guide?

A.On HPC Centers at http://centers.hpc.mil/users/heue/USUserGuide.html.

Q. Where can I find the PBS User Guide?

A.MRAP and Hercules: http://www.arl.hpc.mil/docs/pbsUserGuide.html
Pershing: http://www.arl.hpc.mil/docs/pershingPbsGuide.html

Q. Where can I find the Modules User Guide?

A.On the ARL DSRC Web site at http://www.arl.hpc.mil/docs/modulesUserGuide.html

4. SSH

Q. What do I do when I receive a "Host Key Verification Error"?

A.There are several possible causes for this problem:

  1. The known_hosts file in your home directory has been corrupted. To correct this execute the following commands on Pershing:

    rm -R ${HOME}/.ssh
    exit
    (log back into Pershing from your desktop)
  2. You are using the wrong version of ssh. There are two versions available on Pershing, /usr/bin/ssh and /usr/brl/bin/ssh. Your default version should be /usr/bin/ssh. You can determine which is your default version by executing "which ssh". If /usr/brl/bin/ssh is your current default version, then you can change it by adding the following line to your .cshrc or .profile file:

    .cshrc:
    setenv PATH /usr/bin:$PATH

    .profile:
    set PATH="/usr/bin:$PATH";export $PATH

  3. The access to your home directory (/usr/people/username) is too open. For security reasons, if "group" or "world" have write access to your home directory then ssh will not work. Remove the group/world write access from your home directory to correct this problem.
  4. The access to your .ssh directory (/usr/people/username/.ssh) is too open. For security reasons, if "group" or "world" have write access to this directory then ssh will not work. Remove the group/world write access from this directory to correct this problem.
  5. The compute node you are trying to access is not yet in your known_hosts file. This is only a problem when running batch jobs. To avoid this problem add the following to your run script before you invoke the parallel executable:

    Pershing - csh, tcsh shells
    #===================================================================
    foreach host (`cat $PBS_NODEFILE`)
      echo "Working on $host ...."
      /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
    end
    #===================================================================
    Pershing - sh, ksh, bash shells
    #===================================================================
    host=""
    for new_host in `cat $PBS_NODEFILE`
      do
      if [ "$new_host" != "$host" ]
       then
        host=$new_host
        echo "Working on $host ...."
        /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
      fi
    #===================================================================

5. Job Failures

Q. Why is my job not finding my input file and crashing?

A.Since the compute nodes of Pershing are not able to access your files in /archive or on the Center-Wide File System ($CENTER), you must pre-stage (manually copy) your input files to your /usr/people area ($HOME), space permitting, or to your /usr/var/tmp area ($WORKDIR) before submitting your jobs. Job scripts will need to be modified to pick up input files from these file systems.

Q. Why am I getting a "Segmentation Violation" error in my job?

A.This error generally means that your application has exceeded the stack space limit for the shell in which the job script is running. By default, this value is set to relatively small amount of memory. To correct this problem for csh and tcsh scripts the "unlimit" command should be placed in the .cshrc file in your home directory. For sh and ksh scripts, the "ulimit" command should be placed in the .profile file in your home directory. For bash scripts, the "ulimit" command should be placed in the .bashrc file in your home directory.

Q. Why am I getting a "PBS: job killed: mem NNNNNNNNkb exceeded limit NNNNNNNkb" error in my job?

A.This error means your job is exceeding the maximum available user memory on one or more of the nodes being used by your job. For Pershing, this value is 28 GBytes on the standard memory nodes and 56 GBytes for large memory nodes. For parallel jobs the first option for correcting this problem is to run on more nodes while using fewer processes on each node. For example:

On Pershing, change the PBS option
"select=8,ncpus=16,mpiprocs=16"
to
"select=16,ncpus=16,mpiprocs=8"

If the problem persists, even while using 1 process per node, then you need to redefine your problem to use a smaller memory footprint.

6. COTS Software

Q. I am trying to request a piece of software that is not available, but which has a price for downloading and licensing. What is the process for software of this kind?

A.For all COTS software that we do not have on our systems which will require a purchase, please submit a software request form at: https://reservation.hpc.mil/index-sw_request.html

Q. What software is available on each system at ARL DSRC?

A.Please refer to the software listing at: http://www.arl.hpc.mil/software. Also, performing a "module avail" on the console will display a listing of all COTS applications.

Q. Where can I find an example script for running a COTS package?

A.The $SAMPLES_HOME environment variable points to the directory containing the Sample Code Repository. Execute "ls $SAMPLES_HOME" to see all the sample scripts available for that system. The actual sample scripts are contained within the subdirectories listed. There is also an index file in the main directory explaining the contents of each subdirectory.

Q. How can modules help me?

A. Modules software is recommended for convenience in accessing ARL DSRC COTS software and is available on all our systems. Modules should already be initialized by default for all new users. If you do not see any modules or you get a "module: Command not found." error, you must establish the modules software. To do this, copy the commands, depending on your login shell, from the /usr/cta/modules/samples/ directory into your shell's startup file (.cshrc or .profile and/or .bashrc). Once you have done this, and sourced your .cshrc (or .profile), you can use the module command "module avail" to see what software is available:

Pershing-login02> module avail
------------------------------ /usr/cta/unsupported/COST/modules/pkgs ------------------------------
arpack/9.6(default)         mpi4py/1.3.1(default)       readline/6.2(default)
binutils/2.23(default)      ncar/6.1.2(default)         scalapack/2.0.2(default)
fftw/3.3.3(default)         netcdf/4.3.1.1(default)     scalasca/1.4.2
git/2.0.2(default)          netcdf-mpi/4.3.1.1(default) scalasca/2.0(default)
gnuplot/4.6.1               numpy/1.6.2(default)        scipy/0.11.0(default)
gnuplot/4.6.4(default)      octave/3.2.4(default)       sqlite/3071500(default)
gotoblas/1.13(default)      otf/1.2.6(default)          superlu/3.2
gsl/1.15(default)           papi/5.1.0                  superlu/3.3(default)
hdf5/1.8.12(default)        papi/5.3.0(default)         tau/2.22-p1(default)
hdf5-mpi/1.8.12(default)    parmetis/4.0.2(default)     tcltk/8.6b3(default)
ipython/1.1.0(default)      pcre/8.31(default)          valgrind/3.8.1
lapack/3.4.2(default)       pdt/3.15(default)           valgrind/3.9.0(default)
libxml2/2.9.1               petsc/3.3-p5(default)       xdmf/2013.12.09(default)
matplotlib/1.1.1(default)   pympi/2.5(default)          zlib/1.2.8(default)
matplotlib/1.3.1            python/2.6.8
mpi4py/1.3                  python/2.6.9(default)

----------------------------------- /usr/cta/modules/3.2.7/COTS ------------------------------------
Xpatch/4.10.2(default)            gamess/may13(default)
abaqus/6.12-1(default)            gamess/may13.openmpi
abaqus/6.12-3                     gasp/5.1.2(default)
abaqus/6.13-1                     gaussian
accelrys/6.0(default)             gaussview/5.0.9(default)
accelrys/6.1                      gridgen/15.17(default)
accelrys/7.0                      gridpro/5.1
adf/2012.01(default)              gridpro/5.1_static(default)
adf/2013.01                       gridpro/latest
ale3d/4.16.8(default)             icemcfd/140(default)
ale3d/4.18.8                      ls-dyna/971_R5.1.1
ale3d/4.20.5                      ls-dyna/971_R6.1.0(default)
ale3d/4.22.8                      ls-dyna/971_R7.0.0
ale3d/4.23                        ls-dyna/971_R7.1.1
amber/12(default)                 ls-opt/4.2(default)
ansys/140(default)                ls-prepost/3.2(default)
ansys/145                         ls-prepost/4.0
ansys/150                         maple/16(default)
cart3d/1.3.5(default)             maple/17
castep/6.0                        mathematica/8.0.4(default)
castep/6.1(default)               mathematica/9.0.1
castep/7.0                        matlab/7.14.0(default)
cfd++/10.1                        matlab/8.0.0
cfd++/11.1.1(default)             matlab/8.1.0
cfd++/11.1.1_hpmpi                matlab/8.2.0
cfd++/12.1.1                      mcnpx/v27a(default)
cfd++/12.1.1_hpmpi                mesodyn/6.0(default)
cfd++/12.1.1_intel                mesodyn/6.1
cfd++/14.1                        overflow/2.1ae
cfd++/14.1_hpmpi                  overflow/2.1ae_dp(default)
cfd++/14.1_intel                  paradyn/griz.12.batch.for.paradyn.12
cobalt/5.2(default)               paradyn/griz.12.interactive.for.paradyn.12
cobalt/6.0                        paradyn/griz.13.batch.for.paradyn.13
comsol/4.3b(default)              paradyn/griz.13.interactive.for.paradyn.13
comsol/4.4                        paradyn/paradyn.12.1.12
costinit                          paradyn/paradyn.13.1.15
cseinit                           paradyn/paradyn.13.1.9
cseinit-devel                     paraview/3.98.1
cth/10.2                          paraview/4.0.1
cth/10.2.poe                      paraview/4.0.1_osmesa
cth/10.3(default)                 paraview/4.1.0(default)
cubit/11.1(default)               paraview/4.1.0_osmesa
ddt/3.1(default)                  paraview/4.2.0
ddt/4.1                           paraview/4.2.0_osmesa
discover/6.0(default)             pointwise/17.0r1(default)
discover/6.1                      pointwise/17.1r2
discover/7.0                      pointwise/17.1r3
dmol3/6.0(default)                starccm/8.02.008(default)
dmol3/6.1                         starccm/8.04.007
dmol3/7.0                         starccm/8.04.007-R8
ensight/10.0.2e                   starccm/9.02.005
ensight/10.0.2g                   starccm/9.02.005-R8
ensight/10.0.3c(default)          starccm/9.04.009
ensight/10.1.1b                   starccm/9.04.009-R8
epic/11(default)                  swtestsuite/1(default)
ezviz/1.6                         tecplot/360_2012r1(default)
fluent/13                         tecplot/360_2013r1
fluent/14(default)                tecplot/360_2014r1
fluent/145                        totalview/8.10(default)
fluent/150                        truegrid/2.3.4(default)
g09/a01                           velodyne/2.405
g09/b01                           velodyne/2.502
g09/c01(default)                  velodyne/3.9(default)
g09/c01_bigmem                    visit/2.6.0
g09/d01                           visit/2.6.2
g09/d01_bigmem                    visit/2.7.0
gambit/2.4.6(default)             visit/2.7.1(default)

-------------------------------- /usr/cta/modules/3.2.7/unsupported --------------------------------
DXA/1.3.5(default)           expat/2.0.1(default)         ncar/6.0.0_intel
ImageMagick/6.8.3-0(default) ezviz/1.5(default)           ncl/6.0.0(default)
MOLPRO/2012.1(default)       fftw/2.1.5                   ncl/6.1.2
R/2.13.2(default)            fftw/2.1.5_single            nwchem/6.1(default)
abinit/6.12.3(default)       fftw/3.3.2                   nwchem/6.3
bison/2.4(default)           fftw/3.3.2_gcc               nwchem/6.5
boost/1.53(default)          fftw/3.3.3_gcc               nwchem6.1
cantera/2.0.2(default)       franc3d/6.0.5(default)       openbabel/2.1.5
check_license                geany/1.23.1(default)        openfoam/2.1.0
converge/2.1.0(default)      gnuplot/4.6.3                openfoam/2.1.1(default)
converge/2.1.0_old           gromacs/4.5.5(default)       openfoam/2.3.0
create                       joe/3.7(default)             qe/5.0.1(default)
csd/2014                     lammps/aug12(default)        silo/4.8(default)
csd-2013                     lammps/aug12reax             storm/2.4
dftb+/1.2.2(default)         lammps/aug12shck             texlive/2013(default)
eclipse/4.3.2                namd/2.9(default)            xv/3.10a(default)
epic/11(default)             ncar/6.0.0

----------------------------------- /usr/cta/modules/3.2.7/devel -----------------------------------
compiler/gcc/4.4             mpi/ibmpe/1.2.0.9            mpi/intelmpi/4.1.0.old
compiler/gcc/4.8.2           mpi/ibmpe/1.2.0.9.broken     mpi/intelmpi/4.1.3
compiler/intel/12.1.0        mpi/ibmpe/1.3.0.4            mpi/intelmpi/4.1.3.048
compiler/intel/12.1.6        mpi/intelmpi/4.0.1           mpi/intelmpi/4.1.3.049
compiler/intel/13.0.1        mpi/intelmpi/4.0.3           mpi/intelmpi/5.0.0
compiler/intel/13.1.0        mpi/intelmpi/4.0.3.large     mpi/intelmpi/5.0.0.028
compiler/intel/14.0.0        mpi/intelmpi/4.0.3.old       mpi/mpich2/1.4.1
cuda/5.5                     mpi/intelmpi/4.1.0           mpi/openmpi/1.4.5
java/jdk1.6                  mpi/intelmpi/4.1.0.024       mpi/openmpi/1.6.0
java/jdk1.7                  mpi/intelmpi/4.1.0.024.large mpi/openmpi/1.6.3
mpi/ibmpe/1.2.0.0            mpi/intelmpi/4.1.0.030       mpi/openmpi/1.6.5
mpi/ibmpe/1.2.0.7            mpi/intelmpi/4.1.0.large     vtune/13

-------------------------------- /usr/cta/modules/3.2.7/modulefiles --------------------------------
Master SLB    pbs

You can also use the command "module list" to see what modules are currently loaded:

Pershing-login02> module list
Currently Loaded Modulefiles:
    1) modules   2) pbs       3) Master

Use the command "module load" to load a new module (i.e. select a certain package of software):

Pershing-login02> module load gaussian
Pershing-login02> module list
Currently Loaded Modulefiles:
   1) modules   2) pbs       3) Master    4) Gaussian

A thorough discussion of modules, module commands, and their usage is available in the Modules User Guide. Additional information is also found in each HPC System User Guide about the specific modules that are found on each system.

Q. How do I transfer my data files between the scratch, archive, and CWFS file systems for a batch job if the archive file system is not accessible from the compute nodes?

A.A special PBS queue on Pershing has been setup for this purpose, called the transfer queue. This queue allows serial jobs to run up to 24 hours on a login node, which has access to the scratch ($WORKDIR), the CWFS ($CENTER), and /archive ($ARCHIVE_HOME) directories. Jobs may be submitted directly or within computational job scripts after the computation part of the job has completed. An example of how use the transfer queue can be found in $SAMPLES_HOME/Workload_Management/Transfer_Queue.

Q. How can I transfer my ARL data to another DSRC?

A.The mpscp command is provided at all DSRCs to facilitate transferring data files between DSRC sites. The mpscp command provides a parallel multi-streaming capability to significantly decrease transfer times for very large files. Below are examples of how to use this command.

  1. Transferring data from Pershing to Haise at Navy DSRC using 4 parallel streams:

    mpscp -w 4 ${WORKDIR}/output_data.tgz haise.navo.hpc.mil:/scratch/username/output_data.tgz

  2. Transferring data from Haise at Navy to Pershing using 2 parallel streams:

    mpscp -w 2 haise.navo.hpc.mil:/scratch/username/output_data.tgz ${ARCHIVE_HOME}/output_data.tgz

Q. I can't see the stdout/stderr from my job until after the job completes. Is there any way to check on this output during the run?

A.The qpeek command provides this capability. Its usage is as follows:

qpeek JOB_ID

Q. How can I check on the license usage status of COTS packages?

A.The check_license command provides this capability. Below is a description of its use:

For any non-SLB application, the invocation of the command "check_license application-name" will display the featured application and the number of unused licenses. For Example, the command:

check_license starccm

produces the following output:

Available starccm Licenses are as follows:
ccmpdomains: 100
ccmpsuite: 5
stardesign: 6

Since Abaqus, Cobalt, Fluent and MATLAB are part of Software License Buffer (SLB), the invocation of check_license for an SLB application will display not only the number of unused licenses but also jobs that are waiting for the license to become available as well as jobs that have scheduled future reservations for the application. For example, the command

check_license abaqus

will produce the following output:

Available abaqus Licenses are as follows:
abaqus:50  ams:1  aqua:210  cae:6  cfd:210  cosim_acusolve:1  cosim_direct:1  cse:1  design:210  euler_lagrange:1  explicit:144  foundation:210  multiphysics:1  parallel:16234  standard:162  viewer:1

Pending Jobs for abaqus ready to start are as follows:

ID            Host           # Tokens   Releasing (Hr:Min) Runtime (Hr:Min)
---------     -----------    --------   ------------------ ----------------
205561        pershing-l2          16           0:25             17:42
195579.o2     n0021                14           0:13             71:30
205563        pershing-l2          16           0:25             17:42

Pending Jobs for abaqus with future start times are as follows:

ID            Host           # Tokens   Starting  (Hr:Min) Runtime (Hr:Min)
---------     -----------    --------   ------------------ ----------------
abaqus_7557   PERSHING          256          19:32            168:00

Q. How do I check the status of the PBS batch queues?

A.The show_queues command provides this capability.

Pershing-login02> show_queues

QUEUE INFORMATION for Pershing:
               Maximum   Maximum  Jobs     Jobs     Cores    Cores   Queue
Queue Name    Wall Time   Cores  Running  Pending  Running  Pending Running
---------------------------------------------------------------------------
background     24:00:00    n/a       0        0        0         0       Y
standard-long 200:00:00    n/a       0        0        0         0       Y
challenge     168:00:00    n/a       5       26     1728      2192       Y
cots           96:00:00    n/a       1        0      128         0       Y
debug          01:00:00    n/a       1        0        8         0       Y
high           96:00:00    n/a       0        0        0         0       Y
interactive    12:00:00    n/a       0        0        0         0       Y
prepost        24:00:00      1       0        0        0         0       Y
staff         368:00:00    n/a       0        0        0         0       Y
standard       96:00:00    n/a      44      280     2650     30386       Y
test-hw        96:00:00    n/a       0        0        0         0       Y
transfer       24:00:00    n/a       0        0        0         0       Y
urgent         96:00:00    n/a       0        0        0         0       Y
R309308        72:00:00    640       0        0        0         0       N

NODE USAGE INFORMATION:
   Node Type    Cores Available  Cores Running    Cores Free
   iDataPlex         20160           15364           4796

   Batch       nodes with ALL cores available on them: 191
   Reservation nodes with ALL cores available on them: 18
   Big Memory  nodes with ALL cores available on them: 0

System Node Configuration:
   Node-Type     Nodes     Cores    Physical-Memory  Available-Memory
               Available  Per Node     Per Node          Per Node
   LOGIN              8          16         96Gb             78Gb
   Compute         1260          16         32Gb             29Gb
   Batch *          945          16         32Gb             29Gb
   Reservation *    315          16         32Gb             29Gb
   Big Memory  * ** 168          16         64Gb             58Gb

* These nodes are part of the total Compute pool of nodes.
** These nodes are part of the total Batch pool of nodes.

7. Compilers and MPI Suites

Q. What compilers are available on the Linux Clusters?

A.Please type a "module avail" on the system's console and look under the modules listed under /usr/cta/modules/3.2.7/devel. If you get "module: Command not found", please see the question How can modules help me? to determine how to correct this. Detailed compilation, code optimization and compiler settings can be found in the HPC User Guides on the documentation page: http://www.arl.hpc.mil/docs.

Q. What MPI Suites are available on the Linux Clusters?

A.Please type a "module avail" on the system's console and look under the modules listed under /usr/cta/modules/3.2.7/devel and look for the module names that have MPI as part of the name. Most clusters have at least two MPI versions available. Note that if you get "module: Command not found", please see the question How can modules help me? to determine how to correct this. Detailed compilation, code optimization and compiler settings can be found in the HPC User Guides on the documentation page: http://www.arl.hpc.mil/docs.

8. Miscellaneous

Q. How do I remove characters caused by a FTP session from a PC to a Unix System?

A.Using FTP to transfer a text file in binary mode or a tar file from a PC to a Unix system can cause your file to contain many "^M" characters, representing carriage returns, and "^Z" characters. This can cause problems when compiling that file.

To remove these "^M"s and "^Z"s, use the dos2unix command as follows:

dos2unix file_name

Q. Where should I send back my broken or unwanted YubiKey?

A.Please mail it back to the following address:

AFRL DSRC
ATTN: CCAC ACCOUNTS
2435 Fifth St.
WPAFB, OH 45433

Q. How can I reach the helpdesk?

A.Complete contact information is available on the Contact Us page.

Please Note: All unclassified kerberos and systems support questions should be directed to the HPCMP Consolidated Customer Assistance Center (CCAC). For all other inquiries, you may contact the ARL DSRC Helpdesk.

Q. How do I change permission to a file or a directory?

A. There are three basic modes to files and directories:

  • (r)eadable - Value: 4
  • (w)ritable - Value: 2
  • e(x)ecutable - Value: 1

Additionally, each of these modes can be applied to the

  • (u)ser
  • (g)roup
  • (o)thers

The user means you, the person who owns the file or directory.

The group refers to the unix group associated with the file or directory. To find out what groups you belong to type groups at the unix prompt.

The modes follow a hierarchy of user, group, and then others. Using this we can assign three numerical values.

For example: to make a file readable to all, executable to the group and writable to the user, just add the permission values.

  • user = readable + writable = 4 + 2 = 6
  • group = readable + executable = 4 + 1 = 5
  • other = readable = 4

So to change the permission, type:

% chmod 654 filename

The word "other" means everyone else (aka world), and we do not advise users to open this up for security purposes.

If you need to change permission to an entire directory and its files and its subdirectories, you may use the "-R" option. If you need more information you may review the man pages: man chmod.

Q. What is wrong with my backspace key?

A.All systems use different escape characters to Map your keyboard type to the key. The most common way to work around this problem is to put in your .cshrc or .profile a line "stty erase ^h" in most cases and
"stty erase ^?" in others. If this does not work please contact the Helpdesk.

Q. What shells are available?

A.The Bourne shell (sh), Korn shell (ksh), C-shell (csh), T-shell (tcsh), Bash-shell (bash), and Z shell (zsh) are all available as default shells.

The shell establishes your user environment. Your shell functions as both a command interpreter and a programming language. The shell that you are specified to use resides in the /etc/passwd file.

All shells read the /etc/profile file at start up to set system wide environmental variables. Next your system reads individual user environment files depending on your default shell.

For Bourne shell, Korn shell, and Z shell logins, the shell executes /etc/profile and $HOME/.profile, if it exists.

For C shell logins T shell logins, the shell executes /etc/cshrc, $HOME/.cshrc, and $HOME/.login.

For Bash shell logins, the shell executes /etc/profile, $HOME/.profile, if it exists, and the $HOME/.bashrc, if it exists.

The default /etc/profile and /etc/cshrc files print /etc/motd and check for mail.

Shell Startup Files
ScriptDescription
$HOME/.cshrc Initial commands for each csh
$HOME/.login User's login commands for csh
$HOME/.profile User's login commands for sh and ksh
$HOME/.bashrc User's login commands for bash

.kshrc, .bashrc, and .cshrc are read at each invocation of a new shell.

Q. How do I use the Advance Reservation Service to reserve nodes for future jobs?

A.Information on accessing the Advance Reservation Service (ARS) may be found on the Advance Reservations page. An ARS User Guide is also available once you are logged in to the system.

Q. Where can I find instructions for downloading and using the Utility Server SRD software?

A. On the DAAC website at http://daac.hpc.mil/software/SRD/