TORQUE Resource Manager

2.1 Job Submission

• 2.1.1 Multiple Jobs Submission
• 2.1.2 Requesting Resources
• 2.1.3 Requesting Generic Resources
• 2.1.4 Requesting Floating Resources
• 2.1.5 Requesting Other Resources
• 2.1.6 Exported Batch Environment Variables
• 2.1.7 Enabling Trusted Submit Hosts

Job submission is accomplished using the qsub command, which takes a number of command line arguments and integrates such into the specified PBS command file. The PBS command file may be specified as a filename on the qsub command line or may be entered via STDIN.

• The PBS command file does not need to be executable.
• The PBS command file may be piped into qsub (i.e., cat pbs.cmd | qsub)
• In the case of parallel jobs, the PBS command file is staged to, and executed on, the first allocated compute node only. (Use pbsdsh to run actions on multiple nodes.)
• The command script is executed from the user's home directory in all cases. (The script may determine the submission directory by using the $PBS_O_WORKDIR environment variable)
• The command script will be executed using the default set of user environment variables unless the -V or -v flags are specified to include aspects of the job submission environment.

By default, job submission is allowed only on the TORQUE server host (host on which pbs_server is running). Enablement of job submission from other hosts is documented in Configuring Job Submit Hosts.

2.1.1 Multiple Jobs Submission

Sometimes users will want to submit large numbers of jobs based on the same job script. Rather than using a script to repeatedly call qsub, a feature known as job arrays now exists to allow the creation of multiple jobs with one qsub command. Additionally, this feature includes a new job naming convention that allows users to reference the entire set of jobs as a unit, or to reference one particular job from the set.

qsub -t 0-4 job_script
1098.hostname

qstat
1098-0.hostname ...
1098-1.hostname ...
1098-2.hostname ...
1098-3.hostname ...
1098-4.hostname ...

Job arrays are submitted through the -t option to qsub, or by using #PBS -t in your batch script. This option takes a comma-separated list consisting of either a single job ID number, or a pair of numbers separated by a dash. Each of these jobs created will use the same script and will be running in a nearly identical environment.

Versions of TORQUE earlier than 2.3 had different semantics for the -t arguement. In these versions, -t took a single integer number—a count of the number of jobs to be created.

qsub -t 0-99 would be equvalent to qsub -t 100 in torque 2.2

you can also pass comma delimited lists of ids and ranges:

qsub -t 0,10,20,30,40

or

qsub -t 0-50,60,70,80

Each 1098-x job has an environment variable called PBS_ARRAYID, which is set to the value of the array index of the job, so 1098-0.hostname would have PBS_ARRAYID set to 0. This will allow you to create job arrays where each job in the array will perform slightly different actions based on the value of this variable, such as performing the same tasks on different input files. One other difference in the environment between jobs in the same array is the value of the PBS_JOBNAME variable.

Currently, each job in the array shows up when qstat is run. Essentially they are fully independent TORQUE jobs. All normal TORQUE commands will work on the individual jobs. Eventually, as the job arrays are further developed, a single entry in qstat would be displayed which would summarize the job. An additional flag for qstat would be provided that would show the details of an array. Currently the only TORQUE command that operates on an array as a whole is the qdel command. In the previous example, qdel 1098 would delete every job in the array, while qdel 1098-0 would delete just that one job. Support for qhold and qrls on an entire array will be available shortly, and array awareness will be added to all TORQUE commands one at a time.

Please be aware that job arrays are under development and may have bugs, and certainly are not feature complete. If you have any suggestions or bug reports, please bring them to our attention.

We are currently aware of one limitation that causes the creation of large job arrays on TORQUE installations with high job IDs. This is due to a historical file name length limitation. With thousands of similarly named jobs (and for the time being, each of these jobs has its own job file within pbs_server—they do all share a copy of the script file while on the server), the file name hashing algorithm that attempts to create unique short file names encounters limitations. This limitation for job arrays or sharing the job file will eventually be removed.

2.1.2 Requesting Resources

Various resources can be requested at the time of job submission. A job can request a particular node, a particular node attribute, or even a number of nodes with particular attributes. Either native TORQUE resources, or external scheduler resource extensions may be specified. The native TORQUE resources are listed in the following table:

Example 1 (-l nodes)

> qsub -l nodes=12

> qsub -l nodes=2:server+14

> qsub -l nodes=server:hippi+10:noserver+3:bigmem:hippi

> qsub -l nodes=b2005+b1803+b1813

> qsub -l nodes=4:ppn=2

> qsub -l nodes=1:ppn=4

> qsub -l nodes=2:blue:ppn=2+red:ppn=3+b1014

Example 2

> qsub -l mem=200mb /home/user/script.sh

Example 3

> qsub -l nodes=node01,mem=200mb /home/user/script.sh

2.1.3 Requesting Generic Resources

When generic resources have been assigned to nodes using the server's nodes file, these resources can be requested at the time of job submission using the other field. (See the Consumable Generic Resources page in the Moab Workload Manager Administrator's Guide for details on configuration within Moab).

Example 1

> qsub -l other=matlab /home/user/script.sh

2.1.4 Requesting Floating Resources

When floating resources have been set up inside Moab, they can be requested in the same way as generic resources. Moab will automatically understand that these resources are floating and will schedule the job accordingly. (See the Floating Generic Resources page in the Moab Workload Manager Administrator's Guide for details on configuration within Moab.)

Example 2

> qsub -l other=matlab /home/user/script.sh

2.1.5 Requesting Other Resources

Many other resources can be requested at the time of job submission using the Moab Workload Manger. See the Resource Manager Extensions page in the Moab Workload Manager Administrator's Guide for a list of these supported requests and correct syntax.

2.1.6 Exported Batch Environment Variables

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. These variables are listed in the following table:

2.1.7 Enabling Trusted Submit Hosts

By default, only the node running the pbs_server daemon is allowed to submit jobs. Additional nodes can be trusted as submit hosts by taking any of the following steps:

• Set the allow_node_submit server parameter.
  • Allows any host trusted as a compute host to also be trusted as a submit host.
• Set the submit_hosts server parameter (comma-delimited).
  • Allows specified hosts to be trusted as a submit host.
• Use .rhosts to enable ruserok() based authentication.

See Job Submission Host Advanced Config for more information.

NOTE: If allow_node_submit is set, the parameter allow_proxy_user must be set to allow user proxying when submitting/running jobs.

See Also

• Maui Documentation
• qsub wrapper - Allow local checking and modification of submitted jobs