[torqueusers] Limit max jobs submitted

Andrus, Brian Contractor bdandrus at nps.edu
Tue Nov 22 17:53:13 MST 2011


Martin,

Thanks! That looks like what I need. I have a routing queue and folks use qsub rather than msub. So I will limit how many they are allowed to have in the routing queue. 

Hopefully this will do the trick.


Brian

On Nov 22, 2011, at 2:45 PM, Martin Siegert wrote:

> Hi Brian,
> 
> On Tue, Nov 22, 2011 at 09:28:56PM +0000, Andrus, Brian Contractor wrote:
>> They do work, but they do not do what I need.
>> 
>> See when someone submits >100000 array jobs, it fills up the job list that is used to schedule the jobs.
>> 
>> That is MAXJOB tells how many jobs to work with within moab to decide priority and who to run. So if MAXJOB is set to 50000, and someone submits an array of 100000, then 1/2 of their jobs get pulled in and the rest are ignored (for now) by moab.
>> Now along comes supersensitive.user who submits his interactive job, which will sit for way too long because moab isn't even going to schedule it. In fact, moab is ignoring it.
>> 
>> I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.
>> 
>> Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?
>> 
>> 
>> Brian Andrus
>> ITACS/Research Computing
>> Naval Postgraduate School
>> Monterey, California
>> voice: 831-656-6238
> 
> I ran into exactly the same problem a few weeks ago.
> Currently the only way to prevent a user from overloading moab and thus
> preventing it from scheduling jobs in priority order is to 
> 
> 1) set MAXJOB to some value X
> 2) use
> 
> set queue exec max_user_queuable = Y
> 
>   for the execution queues AND additionally set
> 
> set queue rte max_user_queuable = Y
> 
>   for ALL routing queues that route jobs to the relevant execution queues.
> 
> Y must be much smaller than X.
> Unfortunately, there currently is no limit like
> 
> set server max_user_queuable = Y
> 
> which would be a more logical way of preventing this denial-of-service
> attack against moab.
> 
> Cheers,
> Martin
> 
> -- 
> Martin Siegert
> Simon Fraser University



More information about the torqueusers mailing list