[Mauiusers] maui limits? looking for experience

Arnau Bria arnaubria at pic.es
Thu Sep 29 16:22:31 MDT 2011


On Fri, 30 Sep 2011 07:23:52 +1000
Gareth.Williams at csiro.au Gareth.Williams at csiro.au wrote:

Hi Gareth,

> > This really improves maui behaviour. But limiting idle queue was the
> > last thing I wanted to do....
> 
> Idle limits are mostly good.  This mostly limits the number of each
> users jobs that maui will consider in any scheduling cycle so it make
> the scheduling cycle shorter/faster.  It also limits the priority
> accumulated by queued jobs and alleviates 'queue stuffing'. I'd
> recommend idle limits given that maui does not contain a better
> facility to handle such issues.

Yep, a short queue reduces maui stress. I completely agree that.
Seting a limit of 100 jobs per user  leaves a 1k idle queue in normal
behaviour, when many user are running jobs. That's the limit I'd use.


But, as I've never tried this before, let me ask how maui will behave
in this situation:


if the farm is 70%, and I have only two users who have submited jobs
(user A and B). User A has much more priority than user B, so let's say
that the 30% must be filled with 25% of jobs from user A and 5% jobs
from user B, if I have 1000 jobs in queue (500 from A and 500 from B)
IDLE queue will contain 100 jobs of each user, so each scheduling
cycle is going to schedule 200 jobs, is maui going to fill up the farm
respecting our policies (25/5)? or is it going to start 100 jobs from
each user on each scheduling cycle filling up the farm 15% and 15%?

> > If I understand routing queues properly, they send jobs based on job
> > required resources. our jobs do not require any special resource,
> > our users send jobs based on queue name that show time limits. So,
> > I think that routing queues can't help here.
> 
> What is being proposed is that you have a routing queue setup with no
> special resources, just one routing queue per execution queue (but
> make it as fancy as you like - though simple is good).  Put a limit
> on the number of (users) jobs in the execution queue(s) (enough to
> fill the cluster) but allow many jobs in the routing queue(s).  Maui
> only need consider the execution queue so it's job becomes simpler
> and it can be faster.

ok. now I understand.  So, "hide" jobs to maui using routing queues.

> Gareth (who used maui for some time but doesn't now)

I've not said that. I'm just asking for other admin (which much
experience) experience.
 
Many thanks for your reply,
Cheers,
Arnau


More information about the mauiusers mailing list