[Mauiusers] Torque-Maui preemption problem

Lennart Karlsson Lennart.Karlsson at nsc.liu.se
Thu Sep 29 04:49:42 MDT 2005


Hi Zwika,

We also run such a configuration on a few clusters, with 2 parallel
queues, one for high priority and one for low priority. (Write
me a note outside of this mailing list if you want copies of my
configuration.)

It looks like you have misspelled 'PREEMPTIONPOLICY'. Maui seldom
or never tells you if you have spelled something badly or used
the correct keywords in the wrong context or (my latest mistake,
and caused by a recent change in Maui) written "keyword1 = keyword2" with
spaces between keywords and equals sign. I think that this lack of
error logs is one of the major obstactles when working with Maui. Sometimes
you can use the command 'showconfig' to reassure yourself about that Maui
has understood what you wrote, but it does not work for all parts of the
configuration. It would be very nice to have error logs on syntactic and
semantic errors in the Maui configuration. (CRI, this is a hint for you.)

You can remove the Priority setting in Qmgr, it is enough with those
in your QOSCFG lines. I recommend not to use negative numbers in
priorities and to let your two priority setting be more separated from
each other, so long-queued, low-priority jobs have more difficulties
getting past the high-priority jobs in the scheduling queue. That is,
if you always want your high-priority jobs to have an advantage to
your low-priority jobs. We use 1 for low-priority jobs and 100000 for
high-priority jobs.

I recommend putting 'TIMEOUT=90' on your RMCFG[base] line, so
Maui does not timeout while waiting for Torque. The default timeout value
is possibly not sufficient for the situation you get when Maui is trying
to schedule a high-priority job at some nodes at the same time as Torque
tries to requeue the job already running there. We need this 90 second
timeout for parallel jobs, you might not need it for single-node jobs.

You might want to check that your jobs really are rerunnable, at least
in PBS terms: There is a Rerunable field in the 'qstat -f' output.

I have here given some advice, based on my experiences. But I am not
sure that I have hit on the thing that creates your problems. If you
still have problems I recommend that you give a "print server" output
from your Qmgr and that you add also the pbs_server logs and the
pbs_mom logs.

Best regards,
-- Lennart Karlsson <Lennart.Karlsson at nsc.liu.se>
   National Supercomputer Centre in Linkoping, Sweden
   http://www.nsc.liu.se


> I have got an installation of Torque 1.2.0 & Maui 3.2.6p11.
> 
> The Maui is configured of 2 parallel queues for high & low priorities as
> follows:
> 
>  
> 
> SERVERHOST              creambo
> 
> ADMIN1                root
> 
> RMCFG[base] TYPE=PBS HOST=creambo EPORT=15004
> 
> SOCKETPROTOCOL=HTTP at RMNMHOST@ NMPORT=12321
> 
> CHARGEPOLICY=DEBITALLWC JOBFAILUREACTION=NONE TIMEOUT=15
> 
> RMPOLLINTERVAL       00:00:02
> 
> SERVERPORT            42559
> 
> SERVERMODE            NORMAL
> 
> LOGFILE               maui.log
> 
> LOGFILEMAXSIZE        10000000
> 
> LOGLEVEL              3
> 
> QUEUETIMEWEIGHT       1
> 
> BACKFILLPOLICY         BESTFIT
> 
> RESERVATIONPOLICY     CURRENTHIGHEST
> 
> NODEALLOCATIONPOLICY  MINRESOURCE
> 
> QOSWEIGHT 1
> 
> CREDWEIGHT 1
> 
> PREEMPTOIONPOLICY REQUEUE
> 
> QOSCFG[hi]  PRIORITY=1000 XFTARGET=100 QFLAGS=PREEMPTOR
> 
> QOSCFG[low] PRIORITY=-100 QFLAGS=PREEMPTEE
> 
> CLASSCFG[long]    QDEF=low
> 
> CLASSCFG[short]   QDEF=hi
> 
>  
> 
>  
> 
> These 2 queues are configured in Qmgr as follows:
> 
>  
> 
> Max open servers: 4
> 
> Qmgr: list queue long
> 
> Queue long
> 
>         queue_type = Execution
> 
>         Priority = 100
> 
>         total_jobs = 4
> 
>         state_count = Transit:0 Queued:4 Held:0 Waiting:0 Running:0
> Exiting:0 
> 
>         max_running = 16
> 
>         enabled = True
> 
>  
> 
> Qmgr: list queue short
> 
> Queue short
> 
>         queue_type = Execution
> 
>         Priority = 1000
> 
>         total_jobs = 0
> 
>         state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0
> Exiting:0 
> 
>         max_running = 16
> 
>         resources_assigned.nodect = 0
> 
>         enabled = True
> 
>  
> 
>  
> 
> Once a high priority job is submitted to a totally busy queue, a low
> priority job is preempted, but there is a problem to restart the
> preempted job once the resource is freed. 
> 
> Such a problem occurs also when using preemption policy of Checkpoint or
> Suspend.
> 
> Following a Maui log of such a problem:
> 
>  
> 
> 09/28 09:00:28 INFO:     16 feasible tasks found for job 72710:0 in
> partition DEFAULT (1 Needed)
> 
> 09/28 09:00:28 INFO:     tasks located for job 72710:  1 of 1 required
> (1 feasible)
> 
> 09/28 09:00:28 MJobStart(72710)
> 
> 09/28 09:00:28 MJobDistributeTasks(72710,base,NodeList,TaskMap)
> 
> 09/28 09:00:28 MAMAllocJReserve(72710,RIndex,ErrMsg)
> 
> 09/28 09:00:28 MRMJobStart(72710,Msg,SC)
> 
> 09/28 09:00:28 MPBSJobStart(72710,base,Msg,SC)
> 
> 09/28 09:00:28 MPBSJobModify(72710,Resource_List,Resource,wild1)
> 
> 09/28 09:00:28 ERROR:    job '72710' cannot be started: (rc: 15044
> errmsg: 'Resource temporarily unavailable'  hostlist: 'wild1')
> 
> 09/28 09:00:28 MPBSJobModify(72710,Resource_List,Resource,1)
> 
> 09/28 09:00:28 ALERT:    cannot start job 72710 (RM 'base' failed in
> function 'jobstart')
> 
> 09/28 09:00:28 WARNING:  cannot start job '72710' through resource
> manager
> 
> 09/28 09:00:28 ALERT:    job '72710' deferred after 2 failed start
> attempts (API failure on last attempt)
> 
> 09/28 09:00:28 MJobSetHold(72710,16,1:00:00,RMFailure,cannot start job -
> RM failure, rc: 15044, msg: 'Resource temporarily unavailable')
> 
> 09/28 09:00:28 ALERT:    job '72710' cannot run (deferring job for 3600
> seconds)
> 
> 09/28 09:00:28 MSysRegEvent(JOBDEFER:  defer hold placed on job '72710'.
> reason: 'RMFailure',0,0,1)
> 
> 09/28 09:00:28 MSysLaunchAction(ASList,1)
> 
> 09/28 09:00:28 ERROR:    cannot start job '72710' in partition DEFAULT
> 
> 09/28 09:00:28 MJobPReserve(72710,DEFAULT,ResCount,ResCountRej)
> 
> 09/28 09:00:28 MJobPReserve(72712,DEFAULT,ResCount,ResCountRej)
> 
>  
> 
>  
> 
> Another job resubmission is failed due to rc=15041.
> 
> I must say that this phenomenon is consistent.
> 
>  
> 
> Is there anybody that has encountered such a problem?
> 
>  
> 




More information about the mauiusers mailing list