[Mauiusers] Help with Maui

Steve Young chemadm at hamilton.edu
Thu Jun 26 06:52:02 MDT 2008


On Jun 25, 2008, at 5:42 PM, Jeremy Mann wrote:

> First, Maui has pretty much solved our problems with submitting 100k+
> jobs. It was simply overloading the stock PBS scheduler. But now I'm
> coming into a (maybe) configuration problem.
>
> I have one user who submits 100k+ jobs. This works fine for the first
> couple thousand jobs, which line up in the queue and begin to execute.
> However after a certain time, the queue is only visibily filled  
> with 15
> jobs. There are still thousands of jobs hidden. I can see this if I  
> run
> qstat multiple times in a row, the queue is repopulated with 15  
> more jobs.
> The jobs only run for about 45 seconds so I'm thinking Maui isn't  
> picking
> up on this?

I had a similar situation with a user submitting 7,000 jobs at a  
time. Like you point out maui can't seem to keep up with scheduling  
all of them. After posting to the list it was suggested that I create  
a routing queue in torque:

create queue physics
set queue physics queue_type = Route
set queue physics acl_group_enable = True
set queue physics route_destinations = pompeii
set queue physics enabled = True
set queue physics started = True

Then for the destination queue pompeii I put in the following rule:

set queue pompeii max_queuable = 50

This setup is working well. Torque manages to keep 50 jobs in the  
pompeii execution queue at all times. Maui is happy since it doesn't  
have to go through thousands of jobs each iteration, which it  
couldn't run anyhow due to lack of resources. (I wish we had  
thousands ;-)).


>
> My second question is probably more important. We run two PBS  
> environments
> on all of our clusters. One is our 'default' high priority queue  
> and the
> second is the 'default' low priority queue. The low priority queue  
> is for
> jobs that run at nice 19 or 20. So we load up the low priority  
> queue with
> niced jobs and don't care how long they take to finish. This leaves  
> the
> high priority queue to process our own grid and MPI jobs.
>
> This has worked fine for awhile, but now I have a user who wants to  
> run a
> few hundred thousand jobs in our low priority queue (see paragraph  
> 1 and
> 2). The stock pbs_sched was simply getting overloaded and would crash.
> This is when I set up a test cluster using PBS/Maui and we haven't  
> had a
> problem (other than the 15 queue limit I spoke of before).
>
> Yesterday, I set up a second Maui to schedule the low priority  
> queue. I
> could submit jobs, check job status, however the jobs would never run.
> This is when I started checking the Maui logs and found checksum  
> errors.
> This is when I discovered the problem. The $PATH environment picks  
> up the
> normally installed Maui and uses its binaries to perform its  
> functions.
> Turns out the checksum error is when the normally installed Maui  
> tries to
> process and query the second low priority queue. I can get around  
> this by
> using the second installed Maui's binaries to query the low priority
> queue. If there is a way to disable this at compilation time, I  
> think my
> problem will go away.
>
> I look forward to any comments or questions!
>


As for having two separate instances of torque/maui I'm not sure why  
you would want to do this? Are they load balanced  or something? One  
instance of torque/maui should be able to handle everything you need  
plus you won't have this confusion of keeping track of multiple  
binaries/installations.

-Steve








More information about the mauiusers mailing list