[torqueusers] Limit number of users running jobs simultaneously?

Gabe Turner gabe at msi.umn.edu
Thu Jun 3 16:52:32 MDT 2010


On Thu, Jun 03, 2010 at 02:30:30PM -0700, Garrick Staples wrote:
> On Thu, Jun 03, 2010 at 11:53:04AM -0700, Brad Cavanagh alleged:
> > On Thu, Jun 3, 2010 at 11:49 AM, Garrick Staples <garrick at usc.edu> wrote:
> > > On Thu, Jun 03, 2010 at 11:45:18AM -0700, Brad Cavanagh alleged:
> > >> Is there a way to limit the number of users running jobs for a given
> > >> queue? One of our software packages has a license where only two users
> > >
> > > set queue <queuename> max_running = X
> > >
> > > You didn't even try :(
> > 
> > That only limits the number of jobs that can be running at one time.
> > It doesn't limit the number of users that can be running jobs at one
> > time.
> 
> Heh, when I first read your email, the thought of limiting by unique username
> was so odd that I assumed that your grasp on English was weak!
> 
> > 
> > Like I said, users can submit an unlimited number of jobs to the
> > queue. However, only two users can have jobs running at one time. One
> > user can have one job running and the other can have 351, but I can't
> > have three users running jobs at one time, but a third user could have
> > jobs waiting in the queue.
> 
> I'm not aware of a way to do this, but I bet the Moab people could impliment
> this for you pretty easily.


We actually do do this (as terrible an idea as it may be) using the
submission filter (see the Torque docs about how to enable the filter).
It's pretty simple, but effective: 


myJobs=0;
maxJobs=10;

for line in $(qstat -u $USER | grep $USER | awk '{print $2}'); do
    myJobs=$((myJobs+1));
    # Reject if max jobs submitted
    if (( $myJobs == $maxJobs )); then
        echo -e "\nJob submission denied: Maximum of $maxJobs submitted.\n" >&2
        exit -1;
    fi
done


Of course, once you do this, your users will simply do what ours have done:
Batch their smaller jobs together into really large jobs.

-- 
Gabe Turner                                             gabe at msi.umn.edu
HPC Systems Administrator,
University of Minnesota
Supercomputing Institute                          http://www.msi.umn.edu


More information about the torqueusers mailing list