Customer Support
ASC MSRC Batch Engine Finds a New Home at ARL
By George Petit, Software Engineer, ARL MSRC/Raytheon
As a result of a recent initiative of the High Performance Computing Modernization Program Office (HPCMPO), all new HPC systems at the ARL MSRC as well as all of the other MSRCs, will use LSF as the batch queuing system beginning in FY05.
 |
| Screen capture of Batch Engine GUI |
This marks a significant change since Grid Engine has been the only batch queuing system used at the ARL MSRC since its inception as an MSRC in 1996.
In the unclassified environment, the new machines that will run LSF include a 2048-processor Linux Networx Intel cluster (JVN), and three testbed machines: a 16-processor SGI Altix (Lightfoot), a 32-processor IBM Opteron (Cage), and the testbed for JVN, a 32-processor Linux Networx Intel cluster (JV).
To help users adjust to this new queuing system, the Application Support Team searched for available software that might help mitigate the problems of creating LSF batch scripts for users unfamiliar with the LSF batch system. Two packages were found within the HPCMPO that could be used for this purpose, the ARL MSRC Grid Engine Script Generator Tool (GEST), and Aeronautical Systems Center (ASC) MSRC’s Batch Engine. After analyzing both packages, Batch Engine was selected as it already provides an LSF script template capability and a greater overall capability than GEST.
Batch Engine is a GUI-based program created using the Python programming language. The user can use an LSF script template of existing vendor packages and modify the template to create a specific job script or begin with just the standard LSF job options to create an LSF template/script for their own applications. Figure 1 depicts a Batch Engine template window. The right side of the window contains the script template within a scrollable sub-window and the left side contains a list of run-variant values that are used for each run. Values on the left are linked to specific locations in the script, allowing the users to modify the script by simply updating the values on the left.
Some of the other important capabilities of Batch Engine are:
- Submit jobs remotely to other ARL MSRC LSF-based systems.
- Remotely monitor the status of submitted jobs.
- Transfer data files to and from a remote system.
- Create and save personal templates that are available only to the template creator.
Batch Engine is available on all of the unclassified ARL MSRC LSF-based systems.
Templates are provided for the most-used vendor packages installed on each particular platform.
Future enhancements include a down-loadable PC distribution of Batch Engine that includes the compiled Batch Engine Python scripts into single executable, and the Kerberos and ssh clients in a self-extracting installer. The Batch Engine software may also be ported to the classified environment.
To invoke Batch Engine, a user must perform the following steps:
- Set the DISPLAY environment variable on the ARL HPC machine to your desktop system address.
- execute “xhost + <hpc_system>.arl.hpc.mil” on your desktop system.
- execute “module load batch_engine” on the ARL HPC system.
- execute “begui” on the ARL HPC system.