[torqueusers] How to set queue's max nodes?

Burkhard Bunk bunk at physik.hu-berlin.de
Sun Jan 27 07:49:27 MST 2013


Hi,

the "nodes" resource has always been string-valued, and this cannot
be changed without breaking current installations. (The docs must be in
error at this point.)
Setting a default is ok, as in

    set queue defq resources_default.nodes = 1:ppn=4

but "resources_min.nodes" and "resources_max.nodes" are considered 
invalid.

As we are on it:
The interpretation of "resources_max.nodect" seems to be tricky.
With torque alone (FIFO scheduler), it used to act on a per-queue level,
but when I introduced Maui, the interpretation changed to a per-job 
limitation. I found this with torque-2.5.x and maui-3.3, no idea whether
it's intentional or a bug.
A more precise handling of limits should be possible with Maui's

    CLASSCFG[<queuename>] ...

and a (space separated) list of settings e.g. for

    MAXJOB   MAXJOBPERUSER
    MAXNODE  MAXNODEPERUSER
    MAXPROC  MAXPROCPERUSER

(see Manual, part 6.2.1), but I haven't tested this so far.

Regards,
Burkhard Bunk.
----------------------------------------------------------------------
  bunk at physik.hu-berlin.de      Physics Institute, Humboldt University
  fax:    ++49-30 2093 7628     Newtonstr. 15
  phone:  ++49-30 2093 7980     12489 Berlin, Germany
----------------------------------------------------------------------

On Sun, 27 Jan 2013, shixing wrote:

> When I set the queue both with nodect and nodes like this:create queue team1
> set queue team1 queue_type = Execution
> set queue team1 resources_max.nodect = 2
> set queue team1 resources_max.nodes = 2
> set queue team1 keep_completed = 2592000
> set queue team1 enabled = True
> set queue team1 started = True
> 
> I can also submit large jobs and all the nodes used by the running jobs exceed
> the resources_max.nodect or resources_max.nodes (here are both 2).
> I submit the jobs 4 times like this:
> echo "sleep 100" | qsub -l nodes=1:ppn=4 -q team1
> 
> And qstat shows the command like this:
> 1661.vkvm161057            STDIN            shubao.sx              0 R team1  
>        
> 1662.vkvm161057            STDIN            shubao.sx              0 R team1  
>        
> 1663.vkvm161057            STDIN            shubao.sx              0 R team1  
>        
> 1664.vkvm161057            STDIN            shubao.sx              0 R team1 
> 
> I use torque 4.1.3 and maui 3.3.1.
> 
> On Sun, Jan 27, 2013 at 3:11 AM, Gustavo Correa <gus at ldeo.columbia.edu> wrote:
>       I misundertood Shixing's original question.
>       I though he wanted to prevent each job to exceed a certain number
>       of nodes,
>       but what he wants to apply the limit to the sum of all jobs in the
>       queue, correct?
>
>       Burkhard is right.
>       "nodect" intent seems to be to limit the number of nodes used by
>       all jobs in a specific queue,
>       whereas  AFAIK "nodes" limits the number of nodes each job can
>       request when submitted to
>       a specific queue, right?
>
>       Things may have changed in recent versions, but "nodes" , with the
>       interpretation above, works for me in Torque 2.4.11, with Maui
>       3.2.6p21.
>
>       However, "nodect", with the interpretation above, doesn't work for
>       me,
>       as Shixing also noted, even if I add "ppn=8" to my qsub command,
>       to request all processors in my nodes, and try to exhaust the
>       available resources and hit the nodect limit.
>       Maybe there is a way to implement what Shixing wants in Maui?
>
>       Quoting the Torque Admin Guide, section 4.1.1, "Queues
>       attributes":
>
>       "nodes  integer Specifies the number of nodes "
>       [Note, integer, not a string.  Not in this context at least.]
>
>       "nodect integer         Sets the number of nodes available. By
>       default, TORQUE will set the number of nodes available to the
>       number of nodes listed in the $TORQUE_HOME/server_priv/nodes file.
>       nodect can be set to be greater than or less than that number.
>       Generally, it is used to set the node count higher than the number
>       of physical nodes in the cluster."
> 
> http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue
>       Attributes.htm
>
>       Admittedly, the Guide wording is not very clear.
>       It could include "on a per queue basis", "on a per job basis", or
>       something the like,
>       to clarify the context.
>       The final sentence in "nodect" sounds a bit awkward.
>       Does it work to set the node count *smaller* than the number
>       of physical nodes?
>       Does this depend on the scheduler configuration?  [pbs_sched,
>       Maui, Moab]
>
>       Somebody from Adaptive could clarify.
>
>       Gus Correa
>
>       On Jan 26, 2013, at 8:31 AM, Burkhard Bunk wrote:
>
>       > Hi,
>       >
>       > I would support that.
>       > The correct form, IMHO, is
>       >
>       > set queue myqueue resources_max.nodect = 10
>       >
>       > Regards,
>       > Burkhard Bunk.
>       >
>       ----------------------------------------------------------------------
>       > bunk at physik.hu-berlin.de      Physics Institute, Humboldt
>       University
>       > fax:    ++49-30 2093 7628     Newtonstr. 15
>       > phone:  ++49-30 2093 7980     12489 Berlin, Germany
>       >
>       ----------------------------------------------------------------------
>       >
>       > On Sat, 26 Jan 2013, André Gemünd wrote:
>       >
>       >> Hi,
>       >> I'm a bit surprised by that. Since when is nodes an integer in
>       Torque? It used to be a string in earlier versions.
>       >>
>       >> Greetings
>       >> André
>       >>
>       >> ----- Ursprüngliche Mail -----
>       >>> Hi Shixing
>       >>> Have you tried:
>       >>> set queue myqueue resources_max.nodes = 10
>       >>> More info:
>       >>>http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue
>       Attributes.htm
>       >>> under "Assigning queue resource limits".
>       >>> I hope this helps,
>       >>> Gus
>       >>> On 01/25/2013 05:44 AM, André Gemünd wrote:
>       >>> > I don't think that setting resources_available.nodect has
>       any
>       >>> > effect.
>       >>> > Have you tried resources_max.nodect?
>       >>> >
>       >>> > Greetings
>       >>> >
>       >>> > ----- Ursprüngliche Mail -----
>       >>> >>
>       >>> >> Hi, all:
>       >>> >> Recently I have set up a cluster with 200 nodes. And this
>       cluster
>       >>> >> is
>       >>> >> designed for serving several team. And I want split the
>       nodes to
>       >>> >> some queues.
>       >>> >> I have set the queue attr like this:
>       >>> >> set queue team1 resources_available.nodect = 3
>       >>> >>
>       >>> >>
>       >>> >> But when I submit a job applying nodes>3, it will also run
>       >>> >> successfully. The submit command is :
>       >>> >> echo "sleep 100" | qsub -l nodes=4 -q team1
>       >>> >>
>       >>> >>
>       >>> >> So how can I set the max nodes for the queues?
>       >>> >> --
>       >>> >> Best wishes!
>       >>> >> My Friend~
>       >>> >> _______________________________________________
>       >>> >> torqueusers mailing list
>       >>> >> torqueusers at supercluster.org
>       >>> >> http://www.supercluster.org/mailman/listinfo/torqueusers
>       >>> >>
>       >>> >
>       >>> _______________________________________________
>       >>> torqueusers mailing list
>       >>> torqueusers at supercluster.org
>       >>> http://www.supercluster.org/mailman/listinfo/torqueusers
>       >>
>       >> --
>       >> André Gemünd
>       >> Fraunhofer-Institute for Algorithms and Scientific Computing
>       >> andre.gemuend at scai.fraunhofer.de
>       >> Tel: +49 2241 14-2193
>       >> /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend
>       >> _______________________________________________
>       >> torqueusers mailing list
>       >> torqueusers at supercluster.org
>       >> http://www.supercluster.org/mailman/listinfo/torqueusers
>       > _______________________________________________
>       > torqueusers mailing list
>       > torqueusers at supercluster.org
>       > http://www.supercluster.org/mailman/listinfo/torqueusers
>
>       _______________________________________________
>       torqueusers mailing list
>       torqueusers at supercluster.org
>       http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> 
> 
> 
> --
> Best wishes!
> My Friend~
> 
>


More information about the torqueusers mailing list