[torqueusers] routing queues

Sam Rash srash at yahoo-inc.com
Wed Jan 10 16:44:21 MST 2007


I read on routing queues, but I guess I don't fully get this:

I have 15 usable queues, do I need 15 routing queues?  (ie if I want 4096
max queuable jobs, set the limit per exec queue to 4096/15)

How do I guarantee the top priority jobs are the ones that make it to the
routing queue? (seems impossible--say a job submitted last to a queue is
highest, it won't get out of routing for a long time since torque can't
compute the priority to figure out it's top priority in its queue)

Thanks,


Sam Rash
srash at yahoo-inc.com
408-349-7312
vertigosr37

-----Original Message-----
From: torqueusers-bounces at supercluster.org
[mailto:torqueusers-bounces at supercluster.org] On Behalf Of Garrick Staples
Sent: Tuesday, January 09, 2007 12:44 PM
To: torqueusers at supercluster.org
Subject: Re: [torqueusers] routing queues

On Mon, Jan 08, 2007 at 03:00:30PM -0800, Sam Rash alleged:
> Hey,
> 
> 
> I asked about this quite some time back, but didn't get to fully
understand
> the solution.  Let's say torque has on the order of 20k+ jobs.  They are
> segmented into queues which each of attributes associated with them-cpus,
> memory, etc.  Most importantly, each queue has a priority.  Assume within
a
> queue, we want FIFO priority.
> 
>  
> 
> So, if we want to keep maui's footprint low (say 4k jobs), how do I use
> routing queues to preserve priorities and make sure maui only gets the top
> 40k relevant ones?
> 
>  
> 
> Further, if my prioritization function in maui were much more complex,
then
> the entire 20k+ jobs would need to be IN maui for it to apply it's
priority
> function to find the top jobs, right?
> 
>  
> 
> More or less, we're hitting the case where we've got far more eligible
jobs
> to execute than nodes, but the order they go in matters immensely; in
> particular assume I can even define a static order of the 20k jobs that
> works, 500 new jobs could come in and may need to be anyway-middle,
bottom,
> or near the top.
> 
>  
> 
> What's the solution (Garrick) presented with routing queues?  I read docs,
> but I got caught up in details and couldn't see the forest for the trees


What I do is to have a routing queue in front of the execution queue,
and put a max_queueable on the execution queue.  Then jobs can safely
backlog into the routing queue.

_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers




More information about the torqueusers mailing list