[Mauiusers] Assigning Nodes to specific nodes

Brian Mendenhall brianm at usc.edu
Thu Oct 13 09:34:39 MDT 2011

On Thu, 2011-10-13 at 11:08 +0000, Jonathan Smale wrote:

> set queue firstgen resources_default.neednodes = 1stgennodes
> set nodes compute-0-0 properties = 1stgennodes

Not sure if you were just trying to be brief or if it was simply a typo,
but here you have said you set the node compute-0-0's properties to
'1stgennodes', and later you have it as 'firstgennodes'. If there is a
difference, then that could be the problem.

> I've found a fair few previous emails about this and have followed their solution without sucess.  I can submit the jobs to the queues but they remain in a queued state, qstat -f gives:-
> [root at che-hydra /]# qstat -f 48691
> Job Id: 48691.che-hydra.bham.ac.uk
>     Job_Name = allenega_p2000_g300_r2.cmd
>     Job_Owner = jsmale at che-hydra.bham.ac.uk
>     resources_used.cput = 529:06:32
>     resources_used.mem = 16872kb
>     resources_used.vmem = 245908kb
>     resources_used.walltime = 529:11:31
>     job_state = R

I do believe that 'job_state = R' means it is running from torque's
perspective... Someone correct me if I'm wrong, please.

>     queue = default

Please note the queue it is in, default, *not* firstgen, this is due to
the definition of the default queue, below I'll comment more ...

> checking job 48691
> State: Running
> Creds:  user:jsmale  group:worth  class:default  qos:DEFAULT

According to Maui, the job is running as well, in class 'default'

> [root at che-hydra]# qmgr -c "p s"
> #
> # Create queues and set their attributes.
> #
> #
> # Create and define queue default
> #
> create queue default
> set queue default queue_type = Execution
> set queue default Priority = 100
> set queue default resources_default.nodes = 1
> set queue default enabled = True
> set queue default started = True

So, I think that here is the root of your problem. You have configured
the first queue that a job will be set to (in the server config below)
is the 'default' queue, but this queue is an execution queue which means
that jobs are going to 'run' in this queue. What I believe you want this
queue to be is a routing queue, with route_destinations of the queues
you want your jobs to run in. For example, my default queue
configuration is thus:

create queue default
set queue default queue_type = Route
set queue default max_queuable = 2000
set queue default resources_default.nodes = 1
set queue default resources_default.walltime = 00:30:00
set queue default route_destinations = quick
set queue default route_destinations += long
set queue default route_destinations += large_long
set queue default route_destinations += large_route
set queue default route_destinations += main_route
set queue default route_waiting_jobs = True
set queue default route_lifetime = 600
set queue default enabled = True
set queue default started = True

Please take note that torque will run the job in the FIRST place that it
*can* run in (based on neednodes, resources_min, resources_max), so you
should also order your queues appropriately.

> #
> # Set server attributes.
> #
> set server scheduling = True
> set server acl_host_enable = False
> set server acl_hosts = che-hydra.bham.ac.uk
> set server default_queue = default

^^^ this is the important line that sets the first queue destination.
All jobs that do not specify a queue name will go through this named

I hope this helps,


Brian Mendenhall
Linux/HPCC Administrator
University of Southern California

More information about the mauiusers mailing list