[torqueusers] Limit max jobs submitted

[David Beer]

----- Original Message -----
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> On Nov 22, 2011, at 4:28 PM, Andrus, Brian Contractor wrote:
> 
> > They do work, but they do not do what I need.
> > 
> > See when someone submits >100000 array jobs, it fills up the job
> > list that is used to schedule the jobs.
> > 
> > That is MAXJOB tells how many jobs to work with within moab to
> > decide priority and who to run. So if MAXJOB is set to 50000, and
> > someone submits an array of 100000, then 1/2 of their jobs get
> > pulled in and the rest are ignored (for now) by moab.
> > Now along comes supersensitive.user who submits his interactive
> > job, which will sit for way too long because moab isn't even going
> > to schedule it. In fact, moab is ignoring it.
> 
> I think your are looking for MAXIJOB (idle jobs). MAXIJOB was
> designed to prevent queue stuffing. For instance on our systems, we
> only allow a maximum of 50 idle jobs, meaning Moab will only
> actively increase priority and make attempts to schedule those jobs
> in the idle queue. Any jobs above the 50 are placed in a blocked
> state, and they are basically ignored.
> 
> http://www.adaptivecomputing.com/resources/docs/mwm/6-0/6.2throttlingpolicies.php#idle
> 

This won't completely reject the jobs, but it will help. It will prevent a user from stuffing the queue with jobs, but it won't get Moab to reject things submitted via qsub. Generally speaking, it is not possible for Moab to do that.

David

> 
> > 
> > I could set MAXJOB to 500000, but that still doesn't prevent a user
> > from submitting too many jobs such that the list that is looked at
> > does not over-fill.
> > 
> > Is there a setting were if someone were to submit >X jobs (array or
> > otherwise), torque/moab will not even allow it in?
> > 
> > 
> > Brian Andrus
> > ITACS/Research Computing
> > Naval Postgraduate School
> > Monterey, California
> > voice: 831-656-6238
> > 
> > 
> > 
> > -----Original Message-----
> > From: torqueusers-bounces at supercluster.org
> > [mailto:torqueusers-bounces at supercluster.org] On Behalf Of
> > glen.beane at gmail.com
> > Sent: Tuesday, November 22, 2011 1:23 PM
> > To: Torque Users Mailing List
> > Cc: torqueusers at supercluster.org
> > Subject: Re: [torqueusers] Limit max jobs submitted
> > 
> > 
> > 
> > On Nov 22, 2011, at 4:20 PM, Lloyd Brown <lloyd_brown at byu.edu>
> > wrote:
> > 
> >> You could try one of the per-queue limits
> >> (http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/4.1queueconfig.php).
> >> In particular:
> >> 
> >> - max_queuable
> >> - max_running
> >> - max_user_queuable
> >> - max_user_run
> >> 
> > 
> > As of TORQUE 2.5 these should work correctly with job arrays
> > 
> >> 
> >> 
> >> Lloyd Brown
> >> Systems Administrator
> >> Fulton Supercomputing Lab
> >> Brigham Young University
> >> http://marylou.byu.edu
> >> 
> >> On 11/22/2011 02:15 PM, Andrus, Brian Contractor wrote:
> >>> All,
> >>> 
> >>> 
> >>> 
> >>> Ok, using torque/moab latest versions.
> >>> 
> >>> 
> >>> 
> >>> Have someone submitting array jobs with t>100000
> >>> 
> >>> This causes much grief.
> >>> 
> >>> 
> >>> 
> >>> How to I limit a user from having X number of total jobs in the
> >>> queue
> >>> (to include each individual array element)?
> >>> 
> >>> 
> >>> 
> >>> I see MAXJOBS, but that doesn’t do it.
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> Brian Andrus
> >>> 
> >>> ITACS/Research Computing
> >>> 
> >>> Naval Postgraduate School
> >>> 
> >>> Monterey, California
> >>> 
> >>> voice: 831-656-6238
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> _______________________________________________
> >>> torqueusers mailing list
> >>> torqueusers at supercluster.org
> >>> http://www.supercluster.org/mailman/listinfo/torqueusers
> >> _______________________________________________
> >> torqueusers mailing list
> >> torqueusers at supercluster.org
> >> http://www.supercluster.org/mailman/listinfo/torqueusers
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torqueusers
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/torqueusers
> 
>  ----------------------
>  Steve Crusan
>  System Administrator
>  Center for Research Computing
>  University of Rochester
>  https://www.crc.rochester.edu/
> 
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> 
> iQEcBAEBAgAGBQJOzBYTAAoJENS19LGOpgqKnqQIAIQxzt3cSjJK5Q8cCgMtlScX
> YDttvfWlVaEh7dDV/zL2/MsbDiYBuTBPXb3xGOqvNcalR8SVYMJrFtRW7AkrwBpK
> RG3c8uzL6yqGnIEirWbEFBrdy0FK7+LRILEEg2zJx5OEX2CePECrXbLqvJwka+PX
> TPdSFNfYayrQ4RUxInUK7/wLI/x1oBoHq39O2yHlvbh+X043epnO2RfmvOqGW3D7
> q1/S+BbIyy+ZR4NM+inXJ9FvOeSRus+ytw4ZKd1KsnX6lyxmb7dGsF9xUWmmrh+G
> pupR/eqOf9YhXzO0vyF2QfAzAAi53qXeziR1jda+gB+Iw8PuKDwtU79MMCMtaAg=
> =8w9h
> -----END PGP SIGNATURE-----
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
> 

-- 
David Beer 
Direct Line: 801-717-3386 | Fax: 801-717-3738
     Adaptive Computing
     1712 S East Bay Blvd, Suite 300
     Provo, UT 84606