[torqueusers] Limit max jobs submitted
siegert at sfu.ca
Tue Nov 22 15:45:30 MST 2011
On Tue, Nov 22, 2011 at 09:28:56PM +0000, Andrus, Brian Contractor wrote:
> They do work, but they do not do what I need.
> See when someone submits >100000 array jobs, it fills up the job list that is used to schedule the jobs.
> That is MAXJOB tells how many jobs to work with within moab to decide priority and who to run. So if MAXJOB is set to 50000, and someone submits an array of 100000, then 1/2 of their jobs get pulled in and the rest are ignored (for now) by moab.
> Now along comes supersensitive.user who submits his interactive job, which will sit for way too long because moab isn't even going to schedule it. In fact, moab is ignoring it.
> I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.
> Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
I ran into exactly the same problem a few weeks ago.
Currently the only way to prevent a user from overloading moab and thus
preventing it from scheduling jobs in priority order is to
1) set MAXJOB to some value X
set queue exec max_user_queuable = Y
for the execution queues AND additionally set
set queue rte max_user_queuable = Y
for ALL routing queues that route jobs to the relevant execution queues.
Y must be much smaller than X.
Unfortunately, there currently is no limit like
set server max_user_queuable = Y
which would be a more logical way of preventing this denial-of-service
attack against moab.
Simon Fraser University
More information about the torqueusers