Skip Nav

PBS User Guide

Table of Contents

1. Introduction

Batch queuing systems are used to control access to the compute nodes of large-scale clusters, such as the systems deployed at the ARL DSRC. Without a queuing system, users could overload systems, resulting in tremendous performance degradation. It is the job of a queuing system to regulate processing on each system to maximize job throughput while not overloading the system. The queuing system will run your job as soon as it can while honoring the following:

  • Meets your resource requests
  • Does not overload systems
  • Runs higher priority jobs first

Batch jobs for Harold and MRAP at the ARL DSRC are submitted utilizing the PBS Professional queuing system. The PBS module should be automatically loaded at startup/login, allowing you access to the PBS commands.

2. Submitting a Batch Job

PBS batch jobs are submitted via the qsub command. The format of this command is:

qsub [ options ] batch_script_file

qsub options may be specified on the command line or embedded in the batch script file by lines beginning with "#PBS". Below is a simple PBS script file.

#!/bin/csh
#  Request maximum wallclock time for the job
#PBS -l walltime=01:00:00

#  select=number_of_nodes,ncpus=cores/node,mpiprocs=MPI_procs/node
#  Total cores requested = number_of_nodes X MPI_procs/node
#PBS -l select=2:ncpus=8:mpiprocs=8

#  Specify how to distribute MPI processes across nodes
#PBS -l place=scatter:excl

#  Request job name
#PBS -N myjob

#  Request the PBS job queue for the job
#PBS -q standard

#  Select Project ID
#PBS -A ARLAP96090ARL

#  Request environment variables be exported from the script
#PBS -V

set JOBID=`echo $PBS_JOBID | cut -f1 -d.`
echo job $JOBID starting at `date` on `hostname`\

# All PBS jobs start in user home directory (/usr/people/username)
# Change directory to the job submission directory
cd  $PBS_O_WORKDIR
echo starting in `pwd`

#  Load compiler/MPI modules (required for MPI jobs)
module load ti06/pgi8.0 ti06/openmpi-1.3

## Get the number of cores
set NSLOTS=`wc -l $PBS_NODEFILE | cut -f1 -d " "`
echo NSLOTS is $NSLOTS

set outfile=output.txt
set TMPD=${TMPDIR}/$JOBID
echo TMPD is $TMPD
if (! -e $TMPD ) mkdir -p $TMPD

set infile=example.in
set input=`pwd`/$infile
cp $input $TMPD
cd $TMPD
ls -l

#  The commands in the following section verify access to all the compute nodes to be used.
#====================================================================
setenv MY_HOSTS `cat ${PBS_NODEFILE} | tr ' ' '\n' `
setenv xTMP_HOSTS `echo ${MY_HOSTS} | sed 's/n[0-9][0-9]*/&\-ib/g' | sed 's/ /\n/g' | sort -u `
setenv xTMP_HOSTS2 `echo ${MY_HOSTS} |  sed 's/ /\n/g' | sort -u `

foreach host ($xTMP_HOSTS)
  echo "Working on $host ...."
  /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
end

foreach host ($xTMP_HOSTS2)
  echo "Working2 on $host ...."
  /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
end
#====================================================================
echo start mpi program on `date`
openmpirun.pbs ${HOME}/mympi.exe i=$infile  >> $outfile
set st=$status
echo end mpi program on `date` with status $st

3. Basic PBS Commands

Below is a list of the basic PBS commands and their functions.

Basic PBS Commands
CommandFunction
qsubSubmit a job
qstatCheck status of a job
qstat -qDisplay current status of all PBS queues
qdelDelete a job
qalterModify job submission options
qholdHold or suspend a job
qrlsRelease or unsuspend a job
tracejobDisplay job accounting data from a completed job
pbsnodesObtain host status of all PBS compute nodes
On the MRAP cluster only
apstat -aView all jobs and their respective core allocation type

For a more detailed description of these commands, view the associated man page on the system.

4. Common Batch Directives

Although PBS has many commands and settings, knowing a basic few of those can allow most anyone to run most jobs.

Set the total wall time to run your job:

#PBS -l walltime=HH:MM:SS

Set the queue:

#PBS -q queue_name

Where queue_name is usually debug, standard, challenge or background. Other queues are available, but they are reserved for special cases and/or staff.

To pass all the environmental variables to the parallel jobs, the following PBS directive must be present.

#PBS -V

Select the total number of nodes (not cores!) (x), and the number of cores to use per node (y):

#PBS -l select=x:ncpus=N:mpiprocs=y

The total amount of cores will be equal to x*y (note that "ncpus=" will always be set to 8 for Harold). An example 32-core select is as follows:

#PBS -l select=4:ncpus=8:mpiprocs=8

On MRAP, to select the amount of cores, you must use this format:

#PBS -l mppwidth=24
#PBS -l mppnppn=12

Where mppwidth is the total number of cores your job will utilize and mppnppn is the number of cores/node to use. The example above will run a 24-core job on two nodes (because MRAP has 12 cores/node).

To set the name of the job:

#PBS -N JOB_NAME

For example:

#PBS -N MPI_JOB_93

Please do not use spaces or non-alphanumeric characters (except for the underscore) in JOB_NAME!

The standard output and standard error output from your job can be redirected into one file (optional):

#PBS -j oe

And will be named JOB_NAME.job_# where JOB_NAME is from the "#PBS -N" directive and job_# is given by PBS when PBS runs your job.

To set your project identifier:

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

You should also add the following PBS directive:

# Don't restart job if it fails
#PBS -r n

More information is available in the qsub man pages or on the Web at http://www.pbsgridworks.com/.

5. PBS Environment Variables

There are many PBS environment variables. One needs only to know a select, few important environment variables to get started as shown below. This table is set up assuming an 16-core (2 node) job on the Harold cluster.

PBS Environment Variables
PBS Variable Typical Usage Example Output
$PBS_JOBID echo ${PBS_JOBID} | cut -f1 -d. 321222
$PBS_O_LOGNAME echo $PBS_O_LOGNAME userid
$PBS_O_WORKDIR cd $ PBS_O_WORKDIR; pwd /usr/people/username/subdir
$PBS_NODEFILE cat ${PBS_NODEFILE} | sort r1i0n8
r1i0n8
r1i0n8
r1i0n8
r1i0n8
r1i0n8
r1i0n8
r1i0n8
r1i0n11
r1i0n11
r1i0n11
r1i0n11
r1i0n11
r1i0n11
r1i0n11
r1i0n11
- echo `wc -l ${PBS_NODEFILE} | cut -f1 -d" "` 16
$PBS_ARRAY_INDEX echo "Now starting JOB ARRAY # ${PBS_ARRAY_INDEX}" Now starting JOB ARRAY #72

Some users might find the additional PBS environment variables listed in the table below useful.

Additional PBS Environment Variables
VariableMeaning
NCPUS Number of threads, defaulting to number of cores, on the vnode.
OMP_NUM_THREADS Same as NCPUS and mainly used for OpenMP.
PBS_ARRAY_ID Identifier for job arrays. Consists of sequence number.
PBS_ARRAY_INDEX Index number of subjob in job array.
PBS_ENVIRONMENT Indicates job type: PBS_BATCH or PBS_INTERACTIVE
PBS_JOBCOOKIE Unique identifier for inter-MOM job-based communication.
PBS_JOBID The job identifier assigned to the job or job array by the batch system.
PBS_JOBDIR Pathname of job-specific staging and execution directory.
PBS_JOBNAME The job name supplied by the user.
PBS_MOMPORT Port number on which this job's MOMs will communicate.
PBS_NODEFILE The filename containing a list of vnodes assigned to the job.
PBS_NODENUM Logical vnode number of this vnode allocated to the job.
PBS_O_HOME Value of HOME from submission environment.
PBS_O_HOST The host name on which the qsub command was executed.
PBS_O_LANG Value of LANG from submission environment.
PBS_O_LOGNAME Value of LOGNAME from submission environment.
PBS_O_MAIL Value of MAIL from submission environment.
PBS_O_PATH Value of PATH from submission environment.
PBS_O_QUEUE The original queue name to which the job was submitted.
PBS_O_SHELL Value of SHELL from submission environment.
PBS_O_SYSTEM The operating system name where qsub was executed.
PBS_O_TZ Value of TZ from submission environment.
PBS_O_WORKDIR The absolute path of directory where qsub was executed.
PBS_QUEUE The name of the queue from which the job is executed.
PBS_TASKNUM The task (process) number for the job on this vnode.
TMPDIR The job-specific temporary directory for this job.

6. PBS Sample Scripts for Common MPI Jobs

Shown below are various sample PBS scripts for common MPI executables.
Note: these sample scripts should also exist in /usr/cta/SCR/Parallel_Environment/MPI_PBS_sample on each machine.

6.1. Harold Samples

MPI Program with Intel Fortran and SGI-MPI on Harold

A sample PBS script to run a 128-core (16-node) MPI program compiled with Intel Fortran and utilizing the SGI-MPI library on HAROLD.

#!/bin/csh

# Sample PBS script to run an MPI program compiled with Intel
# Fortran and SGI-MPI

#PBS -l walltime=01:00:00

# 128 core run using 8 cores/node
# select = # of nodes
# ncpus is ALWAYS set to 8!
# mpiprocs is the amount of cores on each node to use
# This run will use (select)x(mpiprocs) cores = 16*8=128 cores
#PBS -l select=16:ncpus=8:mpiprocs=8

# scatter and exclusive mode access
#PBS -l place=scatter:excl

#set name of job (do not use any spaces!)
#PBS -N MPI_JOB

# set queue: debug, standard, challenge, background
#PBS -q standard

# Join the standard error/out into one file (optional)
#xPBS -j oe

# Don't restart job if it fails
#PBS -r n

# Pass all environment variables to MPI jobs
#PBS -V

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

#############################################################
# change the below input variable to your own filename
#############################################################
set input=picalc.exe


# PBS to LSF environmental variable equivalence settings

setenv JOBID `echo ${PBS_JOBID} | cut -f1 -d.`
echo "JOBID,PBS_JOBID=$JOBID,$PBS_JOBID"

setenv LOGNAME ${PBS_O_LOGNAME}
echo "LOGNAME,PBS_O_LOGNAME=$LOGNAME,$PBS_O_LOGNAME"

setenv NSLOTS "`wc -l ${PBS_NODEFILE}| cut -f1 -d' '`"
echo "**** NSLOTS=${NSLOTS}"

echo MPI job $JOBID starting at `date` on `hostname`
echo starting in `pwd`

# exports or setenv

setenv JOBID `echo $PBS_JOBID  | awk -F'.' '{print $1}'`

#setenv MPI_DSM_DISTRIBUTE

# For OpenMP jobs only! (NOT OPENMPI)
#setenv OMP_NUM_THREADS 1

## Get the execution host name
set exechost=`hostname -s`
echo exechost is $exechost

set jobhost=`uname -n`
set outfile=out.dat

set TMPD=/usr/var/tmp/$LOGNAME/$JOBID
set TMPC=/usr/var/tmp/$LOGNAME
if (! -e $TMPD ) mkdir -p $TMPD


# Copy Files to $TMPD which is usually /usr/var/tmp/userid/job #

cp ${input} ${TMPD}
if ( $status != 0 ) then
# Can't find input in $HOME area, then try /usr/var/tmp/userid/ area
   echo "Cannot find $input in `pwd`  area,  trying ${TMPC} area..."
   cd ${TMPC}
   cp ${input} ${TMPD} || ( echo "*ERROR! Cannot cp ${input} ${TMPD} !" ; exit 199 )
endif
cd $TMPD
ls -ltr
unlimit

#  The below section of commands verifies access to all the compute nodes to be used.
#====================================================================
setenv MY_HOSTS `cat ${PBS_NODEFILE} | tr ' ' '\n' `
setenv xTMP_HOSTS `echo ${MY_HOSTS} |  sed 's/ /\n/g' | sort -u `

foreach host ($xTMP_HOSTS)
  echo "Working on $host ...."
  /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
end

#====================================================================
module load compiler/intel11.1 mpi/sgi_mpi-1.26

# REMEMBER! The value after "-np" must be equal to (select)*(mpiprocs)
# in the "PBS -l select=" statement above. In this case it is 128.

mpiexec_mpt -np 128 -v ./picalc.exe >& picalc.out

set st=$status
echo "MPI program ended with status $st on `date`"

exit $st

Job Array on Harold

A sample job script for a JOB ARRAY in PBS on the HAROLD system is:

#!/bin/csh

# Sample PBS script to run an MPI program compiled with Intel
# Fortran and SGI-MPI

#PBS -l walltime=01:00:00
# Job array from 1 to 10 in steps of 1
#PBS -J 1-10:1

# This run will use (select)x(mpiprocs) cores = 16*8=128 cores
#PBS -l select=16:ncpus=8:mpiprocs=8

# scatter and exclusive mode access
#PBS -l place=scatter:excl

#set name of job (do not use any spaces!)
#PBS -N MPI_JOB

# set queue: debug, standard, challenge, background
#PBS -q standard

# Join the standard error/out into one file (optional)
#xPBS -j oe

# Don't restart job if it fails
#PBS -r n

# Pass all environment variables to MPI jobs
#PBS -V

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

#############################################################
# change the below input variable to your own filename
#############################################################
set input=picalc.exe
#

echo "Now starting JOB ARRAY # ${PBS_ARRAY_INDEX} at `date`"

# Environmental variable equivalence settings

setenv JOBID `echo ${PBS_JOBID} | cut -f1 -d.`
echo "JOBID,PBS_JOBID=$JOBID,$PBS_JOBID"

setenv HOSTS `cat ${PBS_NODEFILE} | tr ' ' '\n' `
echo "HOSTS,PBS_NODEFILE=$HOSTS,$PBS_NODEFILE"

setenv LOGNAME ${PBS_O_LOGNAME}
echo "LOGNAME,PBS_O_LOGNAME=$LOGNAME,$PBS_O_LOGNAME"

setenv NSLOTS "`wc -l ${PBS_NODEFILE}| cut -f1 -d' '`"
echo "**** NSLOTS=${NSLOTS}"

echo MPI job $JOBID starting at `date` on `hostname`
echo starting in `pwd`

# exports or setenv

setenv JOBID `echo $PBS_JOBID  | awk -F'.' '{print $1}'`

## Get the execution host name
set exechost=`hostname -s`
echo exechost is $exechost

set jobhost=`uname -n`
set outfile=out.dat
set TMPD=/usr/var/tmp/$LOGNAME/$JOBID
set TMPC=/usr/var/tmp/$LOGNAME
if (! -e $TMPD ) mkdir -p $TMPD

# Copy Files to $TMPD which is usually /usr/var/tmp/userid/job #

cp ${input} ${TMPD}
if ( $status != 0 ) then
# Can't find input in $HOME area, then try /usr/var/tmp/userid/ area
   echo "Cannot find $input in `pwd`  area,  trying ${TMPC} area..."
   cd ${TMPC}
   cp ${input} ${TMPD} || ( echo "*ERROR! Cannot cp ${input} ${TMPD} !" ; exit 199 )
endif
cd $TMPD
ls -ltr
unlimit

#  The below section of commands verifies access to all the compute nodes to be used.
#====================================================================
setenv MY_HOSTS `cat ${PBS_NODEFILE} | tr ' ' '\n' `
setenv xTMP_HOSTS `echo ${MY_HOSTS} |  sed 's/ /\n/g' | sort -u `

foreach host ($xTMP_HOSTS)
  echo "Working on $host ...."
  /usr/bin/ssh -o StrictHostKeyChecking=no $host pwd
end

#====================================================================
module load compiler/intel11.1 mpi/sgi_mpi-1.26

# REMEMBER! The value after "-n" must be equal to (select)*(mpiprocs)
# in the "PBS -l select=" statement above. In this case it is 32.

mpiexec_mpt -np 128 ./picalc.exe >& picalc.out

set st=$status
echo "MPI program ended with status $st on `date`"

exit $st

6.2. MRAP Samples

MPI Program with 16 cores on MRAP

A sample PBS script for running a 16-core MPI job on MRAP:

#!/bin/csh

# Set number of cores to utilize in this run
#PBS -l mppwidth=24

# Set number of cores per node 
#PBS -l mppnppn=12

#Set maximum time of the run
#PBS -l walltime=00:30:00

#PBS -q debug

# Don't use quotes about JOB_NAME or spaces
#PBS -N JOB_NAME_1

#PBS -j oe

# Pass environment variables to mpi job
#PBS -V

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR is set to $PBS_O_WORKDIR"

set TMP=`echo ${PBS_JOBID} | cut -f1 -d.`
set TMPD=$PBS_O_WORKDIR/${TMP}

if ( ! -d ${TMPD} ) then
  mkdir -p $TMPD
endif

cp ./picalc.exe $TMPD || echo "****ERROR! CANNOT cp ./picalc.exe $TMPD "
cd $TMPD

#setenv PAT_RT_SUMMARY 0

echo "Modules in use: --------------vvvvv"
module list
echo "---------------------^^^^^^^^^^^^^^"

echo "Starting at `date` in `pwd` on `hostname`"
aprun -n 24 -N 12 ./picalc.exe
set st=$status

echo "Done at `date` with status $st"
exit $st

Serial Program on MRAP

A sample serial job script for PBS is:

#!/bin/csh

#PBS -l mppwidth=16

# Set number of cores per node 
#PBS -l mppnppn=12

#PBS -l walltime=00:30:00

#PBS -q debug

# Don't use quotes about JOB_NAME
#PBS -N JOB_NAME_1

#PBS -j oe

#PBS -V

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

echo "Now starting at `date`"
cd $PBS_O_WORKDIR

set TMP=`echo ${PBS_JOBID} | cut -f1 -d.`
set TMPD=$PBS_O_WORKDIR/${TMP}
if ( ! -d ${TMPD} ) then
  mkdir -p $TMPD
endif

cp ./picalc.exe $TMPD || echo "**** ERROR! CANNOT cp ./picalc.exe $TMPD"
cd $TMPD

echo "Modules in use: --------------vvvvv"
module list
echo "---------------------^^^^^^^^^^^^^^"

echo "Starting at `date` in `pwd` on `hostname`"
./picalc.exe
set st=$status

echo "Done at `date` with status $st"
exit $st

Job Array on MRAP

A sample job script for a JOB ARRAY in PBS on MRAP is:

#!/bin/csh

#PBS -l mppwidth=24

# Set number of cores per node 
#PBS -l mppnppn=12

#PBS -l walltime=00:30:00

#PBS -q debug

# Don't use quotes about JOB_NAME
#PBS -N JOB_NAME_1

#PBS -j oe

# Job array from 1 to 10 in steps of 1
#PBS -J 1-10:1

#PBS -V

#Set your project alphanumeric (may look like ARLAP96090RAY)
#PBS -A xxxxxxxxxxx

echo "Now starting JOB ARRAY # ${PBS_ARRAY_INDEX} at `date`"

cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR is set to $PBS_O_WORKDIR"

echo "PBS_JOBID=$PBS_JOBID"
set TMP=`echo "${PBS_JOBID}" | cut -f1 -d. | cut -f1 -d[ `
set TMPD=${PBS_O_WORKDIR}/${TMP}
echo "TMPD=${TMPD}"

if ( ! -d ${TMPD} ) then
  mkdir -p $TMPD
endif

cp ./picalc.exe $TMPD || echo "**** ERROR! CANNOT cp ./picalc.exe $TMPD"
cd $TMPD

echo "Modules in use: --------------vvvvv"
module list
echo "---------------------^^^^^^^^^^^^^^"

echo "Starting at `date` in `pwd` on `hostname`"
aprun -n 24 -N 12 ./picalc.exe
set st=$status

echo "Done at `date` with status $st"
exit $st

Customers are reminded to use /usr/var/tmp as their scratch space. Do NOT use /tmp or /var/tmp!