[torqueusers] Submission number limits?
steve.traylen at cern.ch
Wed May 7 14:29:57 MDT 2008
2008/5/7 Garrick Staples <garrick at usc.edu>:
> On Wed, May 07, 2008 at 02:53:04PM -0500, Jeremy Mann alleged:
> > Good afternoon all, I have one user that wants to submit roughly 140,000
> > jobs to our queue. We tried it last week and it never worked. It took
> > nearly an hour to submit all of them, then the PBS scheduler would stop
> > responding and give:
> > 05/02/2008 14:39:50;0100; pbs_sched;Req;;Leaving schedule
> > 05/02/2008 14:39:50;0080; pbs_sched;Svr;main;brk point 760373248
> > 05/02/2008 14:39:53;0100; pbs_sched;Req;;Entering Schedule
> > 05/02/2008 14:42:53;0002; pbs_sched;Svr;toolong;alarm call
> > The jobs are quite small and they run for about a minute. Now we're
> > thinking about breaking them up into 100 or 1000 job chunks.
> > I'm curious if the number of jobs being submitted, in our case 140,000, is
> > too large for PBS/Torque to handle.
> > Torque 2.1.2 x86_64 and the built in scheduler (not MAUI)
> The trick is to limit the number of jobs visible to the scheduler by using a
> routing queue to spool jobs into the execution queue.
> So you do something like this:
> create queue spoolq queue_type = Route, route_destinations = execq
> create queue execq queue_type = E, max_queueable=1000
Would at a MAUI level
do the same thing and allow other users a look in while big user is having his
submitted in batches of 100.
> Submit jobs to spoolq and it should handlelarge numbers of jobs.
> Garrick Staples, GNU/Linux HPCC SysAdmin
> University of Southern California
> Please avoid sending me Word or PowerPoint attachments.
> See http://www.gnu.org/philosophy/no-word-attachments.html
> torqueusers mailing list
> torqueusers at supercluster.org
More information about the torqueusers