[torqueusers] routing queues
srash at yahoo-inc.com
Wed Jan 10 16:44:21 MST 2007
I read on routing queues, but I guess I don't fully get this:
I have 15 usable queues, do I need 15 routing queues? (ie if I want 4096
max queuable jobs, set the limit per exec queue to 4096/15)
How do I guarantee the top priority jobs are the ones that make it to the
routing queue? (seems impossible--say a job submitted last to a queue is
highest, it won't get out of routing for a long time since torque can't
compute the priority to figure out it's top priority in its queue)
srash at yahoo-inc.com
From: torqueusers-bounces at supercluster.org
[mailto:torqueusers-bounces at supercluster.org] On Behalf Of Garrick Staples
Sent: Tuesday, January 09, 2007 12:44 PM
To: torqueusers at supercluster.org
Subject: Re: [torqueusers] routing queues
On Mon, Jan 08, 2007 at 03:00:30PM -0800, Sam Rash alleged:
> I asked about this quite some time back, but didn't get to fully
> the solution. Let's say torque has on the order of 20k+ jobs. They are
> segmented into queues which each of attributes associated with them-cpus,
> memory, etc. Most importantly, each queue has a priority. Assume within
> queue, we want FIFO priority.
> So, if we want to keep maui's footprint low (say 4k jobs), how do I use
> routing queues to preserve priorities and make sure maui only gets the top
> 40k relevant ones?
> Further, if my prioritization function in maui were much more complex,
> the entire 20k+ jobs would need to be IN maui for it to apply it's
> function to find the top jobs, right?
> More or less, we're hitting the case where we've got far more eligible
> to execute than nodes, but the order they go in matters immensely; in
> particular assume I can even define a static order of the 20k jobs that
> works, 500 new jobs could come in and may need to be anyway-middle,
> or near the top.
> What's the solution (Garrick) presented with routing queues? I read docs,
> but I got caught up in details and couldn't see the forest for the trees
What I do is to have a routing queue in front of the execution queue,
and put a max_queueable on the execution queue. Then jobs can safely
backlog into the routing queue.
torqueusers mailing list
torqueusers at supercluster.org
More information about the torqueusers