[torqueusers] How to set queue's max nodes?
Burkhard Bunk
bunk at physik.hu-berlin.de
Sun Jan 27 07:49:27 MST 2013
Hi,
the "nodes" resource has always been string-valued, and this cannot
be changed without breaking current installations. (The docs must be in
error at this point.)
Setting a default is ok, as in
set queue defq resources_default.nodes = 1:ppn=4
but "resources_min.nodes" and "resources_max.nodes" are considered
invalid.
As we are on it:
The interpretation of "resources_max.nodect" seems to be tricky.
With torque alone (FIFO scheduler), it used to act on a per-queue level,
but when I introduced Maui, the interpretation changed to a per-job
limitation. I found this with torque-2.5.x and maui-3.3, no idea whether
it's intentional or a bug.
A more precise handling of limits should be possible with Maui's
CLASSCFG[<queuename>] ...
and a (space separated) list of settings e.g. for
MAXJOB MAXJOBPERUSER
MAXNODE MAXNODEPERUSER
MAXPROC MAXPROCPERUSER
(see Manual, part 6.2.1), but I haven't tested this so far.
Regards,
Burkhard Bunk.
----------------------------------------------------------------------
bunk at physik.hu-berlin.de Physics Institute, Humboldt University
fax: ++49-30 2093 7628 Newtonstr. 15
phone: ++49-30 2093 7980 12489 Berlin, Germany
----------------------------------------------------------------------
On Sun, 27 Jan 2013, shixing wrote:
> When I set the queue both with nodect and nodes like this:create queue team1
> set queue team1 queue_type = Execution
> set queue team1 resources_max.nodect = 2
> set queue team1 resources_max.nodes = 2
> set queue team1 keep_completed = 2592000
> set queue team1 enabled = True
> set queue team1 started = True
>
> I can also submit large jobs and all the nodes used by the running jobs exceed
> the resources_max.nodect or resources_max.nodes (here are both 2).
> I submit the jobs 4 times like this:
> echo "sleep 100" | qsub -l nodes=1:ppn=4 -q team1
>
> And qstat shows the command like this:
> 1661.vkvm161057 STDIN shubao.sx 0 R team1
>
> 1662.vkvm161057 STDIN shubao.sx 0 R team1
>
> 1663.vkvm161057 STDIN shubao.sx 0 R team1
>
> 1664.vkvm161057 STDIN shubao.sx 0 R team1
>
> I use torque 4.1.3 and maui 3.3.1.
>
> On Sun, Jan 27, 2013 at 3:11 AM, Gustavo Correa <gus at ldeo.columbia.edu> wrote:
> I misundertood Shixing's original question.
> I though he wanted to prevent each job to exceed a certain number
> of nodes,
> but what he wants to apply the limit to the sum of all jobs in the
> queue, correct?
>
> Burkhard is right.
> "nodect" intent seems to be to limit the number of nodes used by
> all jobs in a specific queue,
> whereas AFAIK "nodes" limits the number of nodes each job can
> request when submitted to
> a specific queue, right?
>
> Things may have changed in recent versions, but "nodes" , with the
> interpretation above, works for me in Torque 2.4.11, with Maui
> 3.2.6p21.
>
> However, "nodect", with the interpretation above, doesn't work for
> me,
> as Shixing also noted, even if I add "ppn=8" to my qsub command,
> to request all processors in my nodes, and try to exhaust the
> available resources and hit the nodect limit.
> Maybe there is a way to implement what Shixing wants in Maui?
>
> Quoting the Torque Admin Guide, section 4.1.1, "Queues
> attributes":
>
> "nodes integer Specifies the number of nodes "
> [Note, integer, not a string. Not in this context at least.]
>
> "nodect integer Sets the number of nodes available. By
> default, TORQUE will set the number of nodes available to the
> number of nodes listed in the $TORQUE_HOME/server_priv/nodes file.
> nodect can be set to be greater than or less than that number.
> Generally, it is used to set the node count higher than the number
> of physical nodes in the cluster."
>
> http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue
> Attributes.htm
>
> Admittedly, the Guide wording is not very clear.
> It could include "on a per queue basis", "on a per job basis", or
> something the like,
> to clarify the context.
> The final sentence in "nodect" sounds a bit awkward.
> Does it work to set the node count *smaller* than the number
> of physical nodes?
> Does this depend on the scheduler configuration? [pbs_sched,
> Maui, Moab]
>
> Somebody from Adaptive could clarify.
>
> Gus Correa
>
> On Jan 26, 2013, at 8:31 AM, Burkhard Bunk wrote:
>
> > Hi,
> >
> > I would support that.
> > The correct form, IMHO, is
> >
> > set queue myqueue resources_max.nodect = 10
> >
> > Regards,
> > Burkhard Bunk.
> >
> ----------------------------------------------------------------------
> > bunk at physik.hu-berlin.de Physics Institute, Humboldt
> University
> > fax: ++49-30 2093 7628 Newtonstr. 15
> > phone: ++49-30 2093 7980 12489 Berlin, Germany
> >
> ----------------------------------------------------------------------
> >
> > On Sat, 26 Jan 2013, André Gemünd wrote:
> >
> >> Hi,
> >> I'm a bit surprised by that. Since when is nodes an integer in
> Torque? It used to be a string in earlier versions.
> >>
> >> Greetings
> >> André
> >>
> >> ----- Ursprüngliche Mail -----
> >>> Hi Shixing
> >>> Have you tried:
> >>> set queue myqueue resources_max.nodes = 10
> >>> More info:
> >>>http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue
> Attributes.htm
> >>> under "Assigning queue resource limits".
> >>> I hope this helps,
> >>> Gus
> >>> On 01/25/2013 05:44 AM, André Gemünd wrote:
> >>> > I don't think that setting resources_available.nodect has
> any
> >>> > effect.
> >>> > Have you tried resources_max.nodect?
> >>> >
> >>> > Greetings
> >>> >
> >>> > ----- Ursprüngliche Mail -----
> >>> >>
> >>> >> Hi, all:
> >>> >> Recently I have set up a cluster with 200 nodes. And this
> cluster
> >>> >> is
> >>> >> designed for serving several team. And I want split the
> nodes to
> >>> >> some queues.
> >>> >> I have set the queue attr like this:
> >>> >> set queue team1 resources_available.nodect = 3
> >>> >>
> >>> >>
> >>> >> But when I submit a job applying nodes>3, it will also run
> >>> >> successfully. The submit command is :
> >>> >> echo "sleep 100" | qsub -l nodes=4 -q team1
> >>> >>
> >>> >>
> >>> >> So how can I set the max nodes for the queues?
> >>> >> --
> >>> >> Best wishes!
> >>> >> My Friend~
> >>> >> _______________________________________________
> >>> >> torqueusers mailing list
> >>> >> torqueusers at supercluster.org
> >>> >> http://www.supercluster.org/mailman/listinfo/torqueusers
> >>> >>
> >>> >
> >>> _______________________________________________
> >>> torqueusers mailing list
> >>> torqueusers at supercluster.org
> >>> http://www.supercluster.org/mailman/listinfo/torqueusers
> >>
> >> --
> >> André Gemünd
> >> Fraunhofer-Institute for Algorithms and Scientific Computing
> >> andre.gemuend at scai.fraunhofer.de
> >> Tel: +49 2241 14-2193
> >> /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend
> >> _______________________________________________
> >> torqueusers mailing list
> >> torqueusers at supercluster.org
> >> http://www.supercluster.org/mailman/listinfo/torqueusers
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torqueusers
>
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
>
>
>
>
> --
> Best wishes!
> My Friend~
>
>
More information about the torqueusers
mailing list