[torqueusers] Limit max jobs submitted

Steve Crusan scrusan at ur.rochester.edu
Tue Nov 22 14:37:13 MST 2011


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Nov 22, 2011, at 4:28 PM, Andrus, Brian Contractor wrote:

> They do work, but they do not do what I need.
> 
> See when someone submits >100000 array jobs, it fills up the job list that is used to schedule the jobs.
> 
> That is MAXJOB tells how many jobs to work with within moab to decide priority and who to run. So if MAXJOB is set to 50000, and someone submits an array of 100000, then 1/2 of their jobs get pulled in and the rest are ignored (for now) by moab.
> Now along comes supersensitive.user who submits his interactive job, which will sit for way too long because moab isn't even going to schedule it. In fact, moab is ignoring it.

I think your are looking for MAXIJOB (idle jobs). MAXIJOB was designed to prevent queue stuffing. For instance on our systems, we only allow a maximum of 50 idle jobs, meaning Moab will only actively increase priority and make attempts to schedule those jobs in the idle queue. Any jobs above the 50 are placed in a blocked state, and they are basically ignored.

http://www.adaptivecomputing.com/resources/docs/mwm/6-0/6.2throttlingpolicies.php#idle


> 
> I could set MAXJOB to 500000, but that still doesn't prevent a user from submitting too many jobs such that the list that is looked at does not over-fill.
> 
> Is there a setting were if someone were to submit >X jobs (array or otherwise), torque/moab will not even allow it in?
> 
> 
> Brian Andrus
> ITACS/Research Computing
> Naval Postgraduate School
> Monterey, California
> voice: 831-656-6238
> 
> 
> 
> -----Original Message-----
> From: torqueusers-bounces at supercluster.org [mailto:torqueusers-bounces at supercluster.org] On Behalf Of glen.beane at gmail.com
> Sent: Tuesday, November 22, 2011 1:23 PM
> To: Torque Users Mailing List
> Cc: torqueusers at supercluster.org
> Subject: Re: [torqueusers] Limit max jobs submitted
> 
> 
> 
> On Nov 22, 2011, at 4:20 PM, Lloyd Brown <lloyd_brown at byu.edu> wrote:
> 
>> You could try one of the per-queue limits 
>> (http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/4.1queueconfig.php).
>> In particular:
>> 
>> - max_queuable
>> - max_running
>> - max_user_queuable
>> - max_user_run
>> 
> 
> As of TORQUE 2.5 these should work correctly with job arrays
> 
>> 
>> 
>> Lloyd Brown
>> Systems Administrator
>> Fulton Supercomputing Lab
>> Brigham Young University
>> http://marylou.byu.edu
>> 
>> On 11/22/2011 02:15 PM, Andrus, Brian Contractor wrote:
>>> All,
>>> 
>>> 
>>> 
>>> Ok, using torque/moab latest versions.
>>> 
>>> 
>>> 
>>> Have someone submitting array jobs with t>100000
>>> 
>>> This causes much grief.
>>> 
>>> 
>>> 
>>> How to I limit a user from having X number of total jobs in the queue 
>>> (to include each individual array element)?
>>> 
>>> 
>>> 
>>> I see MAXJOBS, but that doesn’t do it.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Brian Andrus
>>> 
>>> ITACS/Research Computing
>>> 
>>> Naval Postgraduate School
>>> 
>>> Monterey, California
>>> 
>>> voice: 831-656-6238
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> torqueusers mailing list
>>> torqueusers at supercluster.org
>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>> _______________________________________________
>> torqueusers mailing list
>> torqueusers at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torqueusers
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers

 ----------------------
 Steve Crusan
 System Administrator
 Center for Research Computing
 University of Rochester
 https://www.crc.rochester.edu/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJOzBYTAAoJENS19LGOpgqKnqQIAIQxzt3cSjJK5Q8cCgMtlScX
YDttvfWlVaEh7dDV/zL2/MsbDiYBuTBPXb3xGOqvNcalR8SVYMJrFtRW7AkrwBpK
RG3c8uzL6yqGnIEirWbEFBrdy0FK7+LRILEEg2zJx5OEX2CePECrXbLqvJwka+PX
TPdSFNfYayrQ4RUxInUK7/wLI/x1oBoHq39O2yHlvbh+X043epnO2RfmvOqGW3D7
q1/S+BbIyy+ZR4NM+inXJ9FvOeSRus+ytw4ZKd1KsnX6lyxmb7dGsF9xUWmmrh+G
pupR/eqOf9YhXzO0vyF2QfAzAAi53qXeziR1jda+gB+Iw8PuKDwtU79MMCMtaAg=
=8w9h
-----END PGP SIGNATURE-----


More information about the torqueusers mailing list