TORQUE Resource Manager

4.1 Queue Configuration

• 4.1.1   Queue Attributes
• 4.1.2   Example Queue Configuration
• 4.1.3   Setting a Default Queue
• 4.1.4   Mapping a Queue to a Subset of Resources
• 4.1.5   Creating a Routing Queue

Under TORQUE, queue configuration is accomplished using the qmgr command.  With this tool, the first step is to create the queue.  This is accomplished using the create subcommand of qmgr as in the following example:

> qmgr -c "create queue batch queue_type=execution"

Once created, the queue must be configured to be operational.  At a minimum, this includes setting the options started and enabled.  Further configuration is possible using any combination of the attributes listed in the section below.

For boolean attributes, T, t, 1, Y, and y are all synonymous with true, and F, f, 0, N, and n all mean false.

For queue_type, E and R are synonymous with Execution and Routing.

4.1.1 Queue Attributes

> qmgr -c "set queue batch acl_groups=staff"
> qmgr -c "set queue batch acl_groups+=ops@h2"
> qmgr -c "set queue batch acl_groups+=staff@h3"

> qmgr -c "set queue batch acl_users=john"
> qmgr -c "set queue batch acl_users+=steve@h2"
> qmgr -c "set queue batch acl_users+=stevek@h3"

> qmgr -c "set queue route route_destinations=fast"
> qmgr -c "set queue route route_destinations+=slow"

Resources may include one or more of the following: arch, mem, nodes, ncpus, nodect, pvmem, and walltime

4.1.2   Example Queue Configuration

qmgr -c "create queue batch queue_type=execution"
qmgr -c "set queue batch started=true"
qmgr -c "set queue batch enabled=true"
qmgr -c "set queue batch resources_default.nodes=1"
qmgr -c "set queue batch resources_default.walltime=3600"

This queue will accept new jobs and, if not explicitly specified in the job, will assign a nodecount of 1 and a walltime of 1 hour to each job.

4.1.3   Setting a Default Queue

qmgr -c "set server default_queue=batch"

4.1.4   Mapping a Queue to a Subset of Resources

The simplest method is using default_resources.neednodes on an execution queue, setting it to a particular node attribute. Maui/Moab will use this information to ensure that jobs in that queue will be assigned nodes with that attribute. For example, say we have some nodes bought with money from the chemistry department, and some nodes paid by the biology department.

$TORQUE_HOME/server_priv/nodes:
node01 np=2 chem
node02 np=2 chem
node03 np=2 bio
node04 np=2 bio

qmgr:
set queue chem resources_default.neednodes=chem
set queue bio  resources_default.neednodes=bio

this example doesn’t preclude other queues from accessing those nodes. One solution is to use some other generic attribute with all other nodes and queues.

More advanced configurations can be made with standing reservations and QOSes. (See TORQUE Queue to Node Mapping)

4.1.5   Creating a Routing Queue

# routing queue
create queue route
set queue route queue_type = Route
set queue route route_destinations = reg_64
set queue route route_destinations += reg_32
set queue route route_destinations += reg
set queue route enabled = True
set queue route started = True

# queue for jobs using 1-15 nodes
create queue reg
set queue reg queue_type = Execution
set queue reg resources_min.ncpus = 1
set queue reg resources_min.nodect = 1
set queue reg resources_default.ncpus = 1
set queue reg resources_default.nodes = 1
set queue reg enabled = True
set queue reg started = True

# queue for jobs using 16-31 nodes
create queue reg_32
set queue reg_32 queue_type = Execution
set queue reg_32 resources_min.ncpus = 31
set queue reg_32 resources_min.nodes = 16
set queue reg_32 resources_default.walltime = 12:00:00
set queue reg_32 enabled = True
set queue reg_32 started = True

# queue for jobs using 32+ nodes
create queue reg_64
set queue reg_64 queue_type = Execution
set queue reg_64 resources_min.ncpus = 63
set queue reg_64 resources_min.nodes = 32
set queue reg_64 resources_default.walltime = 06:00:00
set queue reg_64 enabled = True
set queue reg_64 started = True

# have all jobs go through the routing queue
set server default_queue = batch
set server resources_default.ncpus = 1
set server resources_default.walltime = 24:00:00
  ...

In this example, the compute nodes are dual processors and default walltimes are set according to the number of processors / nodes of a job. Jobs with 32 nodes (63 processors) or more will be given a default walltime of 6 hours.  Also, jobs with 16-31 nodes (31-62 processors) will be given a default walltime of 12 hours.  All other jobs will have the server default walltime of 24 hours.

The ordering of the route_destinations is important.  In a routing queue, a job is assigned to the first possible destination queue based on the resources_max, resources_min, acl_users, and acl_groups attributes.  In the above example, the attributes of a single processor job would first be checked against the reg_64 queue, then the reg_32 queue and finally the reg queue.

Adding the following settings to the above configuration elucidates the queue resource requirements:

set queue reg resources_max.ncpus = 30
set queue reg resources_max.nodect = 15

set queue reg_16 resources_max.ncpus = 62
set queue reg_16 resources_max.ncpus = 31

The time of enforcement of server and queue defaults is important in this example. TORQUE applies server and queue defaults differently in job centric and queue centric modes. For job centric mode, TORQUE waits to apply the server and queue defaults till the job is assigned to its final execution queue. For queue centric mode, it enforces server defaults first, before it is placed in the routing queue. In either mode, queue defaults override the server defaults. TORQUE defaults to job centric mode. To set queue centric mode, set queue_centric_limits, as in:

set server queue_centric_limits = true

An artifact of job centric mode is that if a job does not have an attribute set, the server and routing queue defaults are not applied when queue resource limits are checked. Consequently, a job which requests 32 nodes (not ncpus=32) will not be checked against a min_resource.ncpus limit. Also, for the example above, a job without any attributes set will be placed in the reg_64 queue, since the server ncpus default will be applied after the job is assigned to an execution queue.

Routine queue defaults are NOT applied to job attributes in versions 2.1.0 and before.

If the error message 'qsub: Job rejected by all possible destinations' is reported when submitting a job, it may be necessary to add queue location information, (i.e., in the routing queue's route_destinations attribute, change 'batch' to 'batch@localhost')

See Also

• Server Parameters
• qalter - command which can move jobs from one queue to another