 |
TORQUE Resource Manager |
2.1 Job Submission
Job submission is accomplished using the qsub command. This command takes a number of command line arguments and integrates this into the specified PBS command file. The PBS command file may be specified as a filename on the qsub command line or may be entered via STDIN.- The PBS command file does not need to be executable.
- The PBS command file may be piped into qsub (i.e., 'cat pbs.cmd | qsub')
- In the case of parallel jobs, the PBS command file is staged to, and executed on the first allocated compute node only. (use pbsdsh to run actions on multiple nodes)
- The command script is executed from the user's home directory in all cases (the script may determine the submission directory by using the $PBS_O_WORKDIR environment variable)
- The command script will be executed using the default set of user environment variables unless the '-V' or -v flags are specified to include aspects of the job submission environment.
 |
By default, job submission is allowed only on the TORQUE server host (host on which pbs_server is running). Enablement of job submission from other hosts is documented in Configuring Job Submit Hosts. |
2.1.1 Requesting Resources
Various resources can be requested at the time of job submission. A job can request a particular node, a particular node attribute, or even a number of nodes with particular attributes. Either native TORQUE resources, or external scheduler resource extensions may be specified. The native TORQUE resources are listed in the table below :| Resource | Format | Description |
|---|---|---|
| arch | string | Specifies the administrator defined system architecture required. This defaults to whatever the PBS_MACH string is set to in "local.mk". |
| cput | seconds, or [[HH:]MM:]SS | Maximum amount of CPU time used by all processes in the job |
| file | size* | The amount of total disk requested for the job |
| host | string | Name of the host on which the job should be run. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent. |
| mem | size* | Maximum amount of physical memory used by the job |
| nice | integer between -20 (highest priority) and 19 (lowest priority) | Adjust the process' execution priority |
| nodes | {<node_count> | <hostname>} [:ppn=<ppn>][:<property>[:<property>]...] [+ ...] |
Number and/or type of nodes to be reserved for exclusive use by the job. The value is one or more node_specs joined with the '+' character, "node_spec[+node_spec...]". Each node_spec is an number of nodes required of the type declared in the node_spec and a name or one or more property or properties desired for the nodes. The number, the name, and each property in the node_spec are separated by a colon ':'. If no number is specified, one (1) is assumed.
The name of a node is its hostname. The properties of nodes are:
NOTE: By default, the node resource is mapped to a virtual node or in other words, maps directly to a processor, not a full physical compute node. This behavior can be changed within Maui or Moab by setting the JOBNODEMATCHPOLICY parameter. |
| opsys | string | Specifies the administrator defined system operating system as defined in the mom config file. |
| other | string | Allows a user to specify site specific information. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent. |
| pcput | seconds, or [[HH:]MM:]SS | Maximum amount of CPU time used by any single process in the job |
| pmem | size* | Maximum amount of physical memory used by any single process of the job |
| pvmem | size* | Maximum amount of virtual memory used by any single process in the job |
| software | string | Allows a user to specify software required by the job. This is useful if certain software packages are only available on certain systems in the site. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent. (see Scheduler License Management) |
| vmem | size* | Maximum amount of virtual memory used by all concurrent processes in the job |
| walltime | seconds, or [[HH:]MM:]SS | Maximum amount of real time during which the job can be in the running state |
The size format specifies the maximum amount in terms of bytes or words. It is expressed in the form integer[suffix]. The suffix is a multiplier defined in the following table ('b' means bytes (the default) and 'w' means words). The size of a word is calculated on the execution server as its word size.
| Suffix | Multiplier | |
|---|---|---|
| b | w | 1 |
| kb | kw | 1024 |
| mb | mw | 1,048,576 |
| gb | gw | 1,073,741,824 |
| tb | tw | 1,099,511,627,776 |
Example 1 (-l nodes)
| Usage | Description |
|---|---|
qsub -l
> qsub -l nodes=12
|
request 12 nodes of any type |
qsub -l
> qsub -l nodes=2:server+14
|
request 2 "server" nodes and 14 other nodes (a total of 16) - this specifies two node_specs, "2:server" and "14" |
qsub -l
> qsub -l nodes=server:hippi+10:noserver+3:bigmem:hippi
|
request (a) 1 node that is a "server" and has a "hippi" interface, (b) 10 nodes that are not servers, and (c) 3 nodes that have a large amount of memory an have hippi |
qsub -l
> qsub -l nodes=b2005+b1803+b1813
|
request 3 specific nodes by hostname |
qsub -l
> qsub -l nodes=4:ppn=2
|
request 2 processors on each of four nodes |
qsub -l
> qsub -l nodes=1:ppn=4
|
request 4 processors on one node |
qsub -l
> qsub -l nodes=2:blue:ppn=2+red:ppn=3+b1014
|
request 2 processors on each of two blue nodes, three processors on one red node, and the compute node "b1014" |
Example 2
> qsub -l mem=200mb /home/user/script.sh
Example 3
> qsub -l nodes=node01,mem=200mb /home/user/script.sh
2.1.2 Requesting Generic Resources
When generic resources have been assigned to nodes using the server's nodes file, these resources can be requested at the time of job submission using the other field. (See the Consumable Generic Resources page for details on configuration within Moab).Example 1
> qsub -l other=matlab /home/user/script.sh
NOTE: This can also be requested at the time of job submission using the -W x=GRES:matlab flag.
2.1.3 Requesting Floating Resources
When floating resources have been set up inside Moab, they can be requested in the same way as generic resources. Moab will automatically understand that these resources are floating and will schedule the job accordingly. (See the Floating Generic Resources page for details on configuration within Moab).Example 2
> qsub -l other=matlab /home/user/script.sh
The above job will run on any node when there are enough floating resources available.
|
This can also be requested at the time of job submission using the -W x=GRES:matlab flag. |
2.1.4 Requesting Other Resources
Many other resources can be requested at the time of job submission using the Moab Workload Manger. See the Resource Manager Extensions page for a list of these supported requests and correct syntax.2.1.5 Exported Batch Environment Variables
When a batch job is started, a number of variables are introduced into the job's environment which can be used by the batch script in making decisions, creating output files, etc. These variables are listed in the table below:| Variable | Description |
|---|---|
| PBS_JOBNAME | user specified jobname |
| PBS_O_WORKDIR | user specified jobname |
| PBS_ENVIRONMENT | N/A |
| PBS_TASKNUM | number of tasks requested |
| PBS_O_HOME | home directory of submitting user |
| PBS_MOMPORT | active port for mom daemon |
| PBS_O_LOGNAME | name of submitting user |
| PBS_O_LANG | language variable for job |
| PBS_JOBCOOKIE | job cookie |
| PBS_NODENUM | node offset number |
| PBS_O_SHELL | script shell |
| PBS_O_JOBID | unique pbs job id |
| PBS_O_HOST | host on which job script is currently running |
| PBS_QUEUE | job queue |
| PBS_NODEFILE | file containing line delimited list on nodes allocated to the job |
| PBS_O_PATH | path variable used to locate executables within job script |
See Also
- Many of these commands are also supported in Maui. See the Maui Documentation for details.
- qsub wrapper - Allow local checking and modification of submitted jobs