[Mauiusers] problem with certain jobs blocking others from running

Paul Raines raines at nmr.mgh.harvard.edu
Mon Mar 14 13:47:57 MDT 2011


I have a 127 node cluster running Rocks 5.0 with Torque 2.3.0 and
Maui 3.2.6 (yes, I realize this is pretty old).  Each node has dual Quad
Core Xeon cpus and 32GB of RAM.

I have 16 nodes that have a NVIDIA Tesla GPU installed in them.  I want
only jobs that need the GPU to run on those nodes, and only one job at a time
since the GPU can only be owned by one process at a time.

So I defined a property 'GPU' on all the nodes with GPUs and a property
'nonGPU' on all nodes without.  I created a queue called 'GPU' defined with
resources_default.neednodes = GPU.  And on all our other queues I defined
resources_default.neednodes = nonGPU

I have told all our users running GPU jobs to submit with

    -q GPU -l nodes=1:ppn=8:GPU

The problem I am having is that when

   1) all the GPU nodes are busy
   2) there are more GPU jobs queued
   3) all non-GPU nodes have at least one jobs running on them
 	but still have free processor slots (not job-exclusive)
   4) there are nonGPU jobs to run at lower priority than the GPU jobs
 	waiting to run

the nonGPU jobs waiting do not get run even though there are plenty
of free processr slots still on the nonGPU nodes.

For example here is a section from running showq that shows this.  The 
'mreuter' jobs are nonGPU ones.  The 'rge21' jobs are GPU ones.  You see the 
'mreuter' jobs 1154576 and up are not being run even though there are plenty 
of free nonGPU CPUs.

# showq
.....
1154571 mreuter Running 1 3:23:33:50 Thu Mar 10 17:31:06
1154572 mreuter Running 1 3:23:33:50 Thu Mar 10 17:31:06
1154573 mreuter Running 1 3:23:33:50 Thu Mar 10 17:31:06
1154575 mreuter Running 1 3:23:33:50 Thu Mar 10 17:31:06

    493 Active Jobs 602 of 984 Processors Active (61.18%)
                        123 of  123 Nodes Active      (100.00%)

IDLE JOBS----------------------
JOBNAME            USERNAME      STATE  PROC     WCLIMIT            QUEUETIME

1153960               rge21       Idle     8     2:00:00  Thu Mar 10 12:12:46
1153961               rge21       Idle     8     2:00:00  Thu Mar 10 12:12:47
1153962               rge21       Idle     8     2:00:00  Thu Mar 10 12:12:49
1153963               rge21       Idle     8     2:00:00  Thu Mar 10 12:12:53
1154576             mreuter       Idle     1  4:00:00:00  Thu Mar 10 16:24:36
1154577             mreuter       Idle     1  4:00:00:00  Thu Mar 10 16:24:50
1154578             mreuter       Idle     1  4:00:00:00  Thu Mar 10 16:25:04
1154579             mreuter       Idle     1  4:00:00:00  Thu Mar 10 16:25:18
....


Now as temp fix for this, I run these two commands:

   showq | grep rge21 | grep Idle | cut -d' ' -f1 | xargs qhold

wait a minute and then run

   showq | grep rge21 | grep Hold | cut -d' ' -f1  | xargs qrls

Then all the nonGPU jobs held up get run.

As I mentioned this only seems to happen when all nonGPU nodes have at least 
one processor assigned to a job. I understand why that would hold up jobs if 
those rge21 jobs where nonGPU jobs.  So it seems maui is not properly taking 
into account the properties of nodes along with the needednodes resource of 
jobs in scheduling.

Is maui simply incapable of dealing with this situation or is there
some config I can use to handle it?

A few lines from maui.cfg that might be relavent.

QUEUETIMEWEIGHT       1
CLASSWEIGHT           10
USERCFG[DEFAULT] MAXIPROC=32
BACKFILLPOLICY        FIRSTFIT
RESERVATIONPOLICY     CURRENTHIGHEST
NODEALLOCATIONPOLICY  PRIORITY
NODECFG[DEFAULT] PRIORITY=1000 PRIORITYF='PRIORITY + .01 * AMEM - 10 * JOBCOUNT - 2 * USAGE'


-- 
---------------------------------------------------------------
Paul Raines                email: raines at nmr.mgh.harvard.edu
MGH/MIT/HMS Athinoula A. Martinos Center for Biomedical Imaging
149 (2301) 13th Street     Charlestown, MA 02129	    USA




The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.



More information about the mauiusers mailing list