[torqueusers] Job array throttling

Gareth.Williams at csiro.au Gareth.Williams at csiro.au
Thu Jul 10 18:18:24 MDT 2008


Hi Gabe,

I suggest that you fix the policy... I can't see a lot of point in
having array jobs only up to a size of ten - and that subject to the
user having no existing queued jobs.

You might look through the archives of the mailing list for info on
setting up per user limits on queued jobs and having routing queues (to
hold as many jobs as you like) that feed into the limited queues.

Cheers,

Gareth Williams
Outreach Manager
CSIRO IM&T - ASC
Ph +61 3 8601 3804 (CSIRO)
Ph +61 3 9669 8114 (BoM)
http://intra.hpsc.csiro.au
calendar: http://tinyurl.com/3mkl5d


> -----Original Message-----
> From: Gabe Turner [mailto:gabe at msi.umn.edu]
> Sent: Friday, 11 July 2008 12:06 AM
> To: torqueusers at supercluster.org
> Subject: [torqueusers] Job array throttling
> 
> We starting to play with job arrays as we have some actual requests
for
> them from former SGE users which we've recently adopted.  Submitting
and
> delting them is working fine (in 2.3.0), but there seems to be no way
that
> I can tell to throttle them.  Basically, I would like to limit the
size of
> any given job array to a static number of jobs.  We have a policy, as
> unwise and politically-motivated as it may be, dictating that no user
can
> submit more than 10 jobs.  Currently we throttle this using a submit
> filter.
> 
> What our submit filter does is just parse qstat output to determine
the
> number of jobs a user has submitted (crude, but it works) and it
returns
> an
> error when they try to submit jobs beyond 10.  The problem is that a
user
> would be able to circumvent this by submitting a job array containing
more
> than 10 jobs, as it seems the submit filter is only executed once,
even
> when sumitting a job array.  Fortunately, once this 10+ job array has
been
> submitted, our filter then prevent further submissions for that user.
> 
> We can work support for detecting '#PBS -t ...' into our submit
filter,
> but
> I don't believe that will work if -t is passed on the qsub command
line.
> I've considered the implementation of a qsub wrapper, but frankly it's
> just
> too easy to circumvent.
> 
> Anyone want to brainstorm with me about this?  Any plans by those
Torque
> developers working on job arrays to providing throttling policies for
job
> array size?
> 
> Any help would be greatly appreciated!
> 
> Gabe
> 
> 
> --
> Gabe Turner
gabe at msi.umn.edu
> UNIX System Administrator,
> University of Minnesota
> Supercomputing Institute
http://www.msi.umn.edu
> 




More information about the torqueusers mailing list