[torqueusers] Limit max jobs submitted

Gareth.Williams at csiro.au Gareth.Williams at csiro.au
Tue Nov 22 20:28:35 MST 2011


> -----Original Message-----
> From: glen.beane at gmail.com [mailto:glen.beane at gmail.com]
> Sent: Wednesday, 23 November 2011 12:25 PM
> To: Torque Users Mailing List
> Subject: Re: [torqueusers] Limit max jobs submitted
> 
> On Nov 22, 2011, at 5:45 PM, Martin Siegert <siegert at sfu.ca> wrote:
> 
> > Hi Brian,
> >
> > On Tue, Nov 22, 2011 at 09:28:56PM +0000, Andrus, Brian Contractor
> wrote:
> >> They do work, but they do not do what I need.
> >>
> >> See when someone submits >100000 array jobs, it fills up the job
> list that is used to schedule the jobs.
> >>
> >> That is MAXJOB tells how many jobs to work with within moab to
> decide priority and who to run. So if MAXJOB is set to 50000, and
> someone submits an array of 100000, then 1/2 of their jobs get pulled
> in and the rest are ignored (for now) by moab.
> >> Now along comes supersensitive.user who submits his interactive job,
> which will sit for way too long because moab isn't even going to
> schedule it. In fact, moab is ignoring it.
> >>
> >> I could set MAXJOB to 500000, but that still doesn't prevent a user
> from submitting too many jobs such that the list that is looked at does
> not over-fill.
> >>
> >> Is there a setting were if someone were to submit >X jobs (array or
> otherwise), torque/moab will not even allow it in?
> >>
> >>
> >> Brian Andrus
> >> ITACS/Research Computing
> >> Naval Postgraduate School
> >> Monterey, California
> >> voice: 831-656-6238
> >
> > I ran into exactly the same problem a few weeks ago.
> > Currently the only way to prevent a user from overloading moab and
> thus
> > preventing it from scheduling jobs in priority order is to
> >
> > 1) set MAXJOB to some value X
> > 2) use
> >
> > set queue exec max_user_queuable = Y
> >
> >   for the execution queues AND additionally set
> >
> > set queue rte max_user_queuable = Y
> >
> >   for ALL routing queues that route jobs to the relevant execution
> queues.
> >
> > Y must be much smaller than X.
> > Unfortunately, there currently is no limit like
> >
> > set server max_user_queuable = Y
> >
> 
> 
> This is a reasonable feature request. I have been thinking about
> implementing it for a few years...

Hi Glen (and all),

I find your statement confusing.  Do you mean you have been thinking about adding this to your torque configuration or and you thinking of implementing a new feature in torque?

On a related note, I posted a while ago that such a routing queue setup with limits does not play nicely with job dependencies.  I'd probably move to such a setup if this issue were resolved.  There would still be a potential issue of a user wanting to submit higher priority jobs when they already have filled their limit.  I guess they have to hold their existing queued jobs if they want to handle such a situation.

It would be nice if the scheduler was more graceful about dealing with many jobs but I can see there are technical issues that make that hard.  We recently had to increase limits in moab to get more jobs considered but then ran out of memory which was a worse issue.

Gareth

> 
> 
> 
> 
> > which would be a more logical way of preventing this denial-of-
> service
> > attack against moab.
> >
> > Cheers,
> > Martin
> >
> > --
> > Martin Siegert
> > Simon Fraser University
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list