[torqueusers] routing queues

[Garrick Staples]

On Mon, Jan 08, 2007 at 03:00:30PM -0800, Sam Rash alleged:
> Hey,
> 
> 
> I asked about this quite some time back, but didn't get to fully understand
> the solution.  Let's say torque has on the order of 20k+ jobs.  They are
> segmented into queues which each of attributes associated with them-cpus,
> memory, etc.  Most importantly, each queue has a priority.  Assume within a
> queue, we want FIFO priority.
> 
>  
> 
> So, if we want to keep maui's footprint low (say 4k jobs), how do I use
> routing queues to preserve priorities and make sure maui only gets the top
> 40k relevant ones?
> 
>  
> 
> Further, if my prioritization function in maui were much more complex, then
> the entire 20k+ jobs would need to be IN maui for it to apply it's priority
> function to find the top jobs, right?
> 
>  
> 
> More or less, we're hitting the case where we've got far more eligible jobs
> to execute than nodes, but the order they go in matters immensely; in
> particular assume I can even define a static order of the 20k jobs that
> works, 500 new jobs could come in and may need to be anyway-middle, bottom,
> or near the top.
> 
>  
> 
> What's the solution (Garrick) presented with routing queues?  I read docs,
> but I got caught up in details and couldn't see the forest for the trees


What I do is to have a routing queue in front of the execution queue,
and put a max_queueable on the execution queue.  Then jobs can safely
backlog into the routing queue.