[torquedev] routing queues

Martin Siegert siegert at sfu.ca
Wed Aug 4 19:58:58 MDT 2010

On Wed, Jul 28, 2010 at 07:21:01PM -0700, Martin Siegert wrote:

> Can somebody tell me which routine in the code is responsible for
> assigning the correct queue?

I believe I now understand at least somewhat how routing is implemented.
Consider the following setup:

set queue default queue_type = Route                                        
set queue default route_destinations = qs                                   
set queue default route_destinations += q1                                  

set queue qs queue_type = Execution                                         
set queue qs resources_min.nodes = 1:ppn=2                                  
set queue qs resources_min.procs = 2                                        

set queue q1 queue_type = Execution                                         
set queue q1 resources_max.nodect = 1                                       
set queue q1 resources_max.nodes = 1:ppn=1                                  
set queue q1 resources_max.procs = 1                                        

First scenario:
Submit a job with -l nodes=1:ppn=1.
In order to route the job into the correct queue the routine comp_resc2
in attr_fn_resc.c is called. This routine is supposed to check whether
the min requirements of the queue are satisfied. When this is checked for
the queue "qs" in the case of *wiresc->rs_defin->rs_name being "nodes" the
rs_comp routine at line 605 (probably also at line 550) basically runs a
strcmp("1:ppn=2", "1:ppn=1") which returns 1 and consequently comp_resc_gt
is increased by 1, hence the check comp_resc_gt > 0 at svr_jobfunc.c,
line 1466 is true and chk_resc_limits returns with PBSE_EXCQRESC indicating
that the min requirements are indeed violated for queue qs. Everything
is correct.

Second scenario:
Submit a job with -l nodes=1:ppn=10.
Everything remains the same! The point being that rs_comp again runs a
strcmp, now with arguments "1:ppn=2", "1:ppn=10", which also returns 1
indicating again that the min requirements are violated for queue qs.
This cannot be correct.

Now swap order of the route destinations:

set queue default route_destinations = q1                                   
set queue default route_destinations += qs

and submit a job with -l nodes=1:ppn=2.                                  
This time the critical code is in the routine chk_svr_resc_limit (in
svr_jobfunc.c) which is called from chk_resc_limits (svr_jobfunc.c,
line 1476). When the nodes resource is checked, i.e., LimitName at line
1023 svr_jobfunc.c is set to "nodes", the comparison between job request
and queue settings are made in lines 1050 - 1058 of svr_jobfunc.c:

          if ((isdigit(*(jbrc_nodes->rs_value.at_val.at_str))) &&
            job_nodes   = atoi(jbrc_nodes->rs_value.at_val.at_str);
            queue_nodes = atoi(qurc->rs_value.at_val.at_str);

            if (queue_nodes < job_nodes)

and jbrc_nodes->rs_value.at_val.at_str is "1:ppn=2" and
qurc->rs_value.at_val.at_str is "1:ppn=1". Hence both
*(jbrc_nodes->rs_value.at_val.at_str) and
*(qurc->rs_value.at_val.at_str) are 1 and both job_nodes and queue_nodes
are set to 1. Consequently no violation of the max. queue limit is
detected. In fact, the whole :ppn=x part is ignored.

In conclusion, routing based on "nodes" requests is not working.
The question is how should it be working? E.g., what is "larger":
nodes=1:ppn=2 or nodes=2:ppn=1? This does not really make sense.
IMHO the only sensible comparision for requests of the form
nodes=x:ppn=y is to compare x*y with queue limits. And such a method
could then be expanded to include procs as well: there should be an
attribute, say procct, that gets set to x*y + z, where z comes from
procs=z requests; procct can be calculated for the queue min and max
limits and compared with the job request. 

- Martin

More information about the torquedev mailing list