[Mauiusers] Torque-Maui preemption problem

Zvika Galant zvika at Camero-Tech.com
Thu Sep 29 06:34:50 MDT 2005


Hi, Lennart

Thanks for the help.
I have adopted all your recommendations, but it still doesn't work.

NOTE: the job has been restarted on a node named wild8, but after
preemption I have checked its exec_host once again (by qstat -f) and
it's different.

According to your request, following the requested cfg and logs,
regarding the rerun problem of job 73499.creambo:


Qmgr's print server:

create queue default
set queue default queue_type = Execution
set queue default enabled = True
set queue default started = True
create queue short
set queue short queue_type = Execution
set queue short max_running = 16
set queue short enabled = True
create queue long
set queue long queue_type = Execution
set queue long max_running = 16
set queue long enabled = True
set server scheduling = True
set server managers = root at creambo
set server operators = root at creambo
set server default_queue = default
set server log_events = 511
set server mail_from = adm
set server scheduler_iteration = 600
set server node_ping_rate = 300
set server node_check_rate = 600
set server tcp_timeout = 6
set server job_stat_rate = 30


pbs_server's log:

09/29/2005 14:15:08;0100;PBS_Server;Req;;Type ModifyJob request received
from root at creambo, sock=9
09/29/2005 14:15:08;0008;PBS_Server;Job;73499.creambo;Job Modified at
request of root at creambo
09/29/2005 14:15:08;0100;PBS_Server;Req;;Type RunJob request received
from root at creambo, sock=9
09/29/2005 14:15:08;0008;PBS_Server;Job;73499.creambo;Job Run at request
of root at creambo
09/29/2005 14:15:08;0008;PBS_Server;Job;73499.creambo;unable to run job,
MOM rejected
09/29/2005 14:15:08;0080;PBS_Server;Req;req_reject;Reject reply
code=15041( MSG=send failed, STARTING), aux=0, type=RunJob, from
root at creambo
09/29/2005 14:15:08;0040;PBS_Server;Svr;creambo;Scheduler sent command
new
09/29/2005 14:15:08;0100;PBS_Server;Req;;Type ModifyJob request received
from root at creambo, sock=9
09/29/2005 14:15:08;0008;PBS_Server;Job;73499.creambo;Job Modified at
request of root at creambo
09/29/2005 14:15:08;0100;PBS_Server;Req;;Type AuthenticateUser request
received from zvika at creambo, sock=13


pbs_mom's log:

09/29/2005 14:16:02;0001;   pbs_mom;Job;TMomFinalizeJob3;job
73499.creambo started, pid = 7769
09/29/2005 14:16:02;0008;   pbs_mom;Job;73499.creambo;Job Modified at
request of PBS_Server at creambo
09/29/2005 14:16:07;0001;   pbs_mom;Job;TMomFinalizeJob3;job
73500.creambo started, pid = 7801
09/29/2005 14:16:07;0008;   pbs_mom;Job;73500.creambo;Job Modified at
request of PBS_Server at creambo
09/29/2005 14:16:16;0008;   pbs_mom;Job;73499.creambo;kill_task: killing
pid 7769 task 1 with sig 9
09/29/2005 14:16:17;0008;   pbs_mom;Job;73499.creambo;kill_task: killing
pid 7784 task 1 with sig 9
09/29/2005 14:16:17;0008;   pbs_mom;Job;73499.creambo;kill_task: killing
pid 7785 task 1 with sig 9
09/29/2005 14:16:17;0008;   pbs_mom;Job;73499.creambo;kill_task: killing
pid 7799 task 1 with sig 9
09/29/2005 14:16:17;0080;
pbs_mom;Job;73499.creambo;scan_for_terminated: job 73499.creambo task 1
terminated, sid 7769
09/29/2005 14:16:17;0008;   pbs_mom;Job;73499.creambo;Terminated
09/29/2005 14:16:17;0008;   pbs_mom;Job;73499.creambo;Job Modified at
request of PBS_Server at creambo


Thanks in advance,

Zvika Galant

-----Original Message-----
From: mauiusers-bounces at supercluster.org
[mailto:mauiusers-bounces at supercluster.org] On Behalf Of Lennart
Karlsson
Sent: Thursday, September 29, 2005 12:50 PM
To: mauiusers at supercluster.org
Subject: Re: [Mauiusers] Torque-Maui preemption problem 

Hi Zwika,

We also run such a configuration on a few clusters, with 2 parallel
queues, one for high priority and one for low priority. (Write
me a note outside of this mailing list if you want copies of my
configuration.)

It looks like you have misspelled 'PREEMPTIONPOLICY'. Maui seldom
or never tells you if you have spelled something badly or used
the correct keywords in the wrong context or (my latest mistake,
and caused by a recent change in Maui) written "keyword1 = keyword2"
with
spaces between keywords and equals sign. I think that this lack of
error logs is one of the major obstactles when working with Maui.
Sometimes
you can use the command 'showconfig' to reassure yourself about that
Maui
has understood what you wrote, but it does not work for all parts of the
configuration. It would be very nice to have error logs on syntactic and
semantic errors in the Maui configuration. (CRI, this is a hint for
you.)

You can remove the Priority setting in Qmgr, it is enough with those
in your QOSCFG lines. I recommend not to use negative numbers in
priorities and to let your two priority setting be more separated from
each other, so long-queued, low-priority jobs have more difficulties
getting past the high-priority jobs in the scheduling queue. That is,
if you always want your high-priority jobs to have an advantage to
your low-priority jobs. We use 1 for low-priority jobs and 100000 for
high-priority jobs.

I recommend putting 'TIMEOUT=90' on your RMCFG[base] line, so
Maui does not timeout while waiting for Torque. The default timeout
value
is possibly not sufficient for the situation you get when Maui is trying
to schedule a high-priority job at some nodes at the same time as Torque
tries to requeue the job already running there. We need this 90 second
timeout for parallel jobs, you might not need it for single-node jobs.

You might want to check that your jobs really are rerunnable, at least
in PBS terms: There is a Rerunable field in the 'qstat -f' output.

I have here given some advice, based on my experiences. But I am not
sure that I have hit on the thing that creates your problems. If you
still have problems I recommend that you give a "print server" output
from your Qmgr and that you add also the pbs_server logs and the
pbs_mom logs.

Best regards,
-- Lennart Karlsson <Lennart.Karlsson at nsc.liu.se>
   National Supercomputer Centre in Linkoping, Sweden
   http://www.nsc.liu.se


> I have got an installation of Torque 1.2.0 & Maui 3.2.6p11.
> 
> The Maui is configured of 2 parallel queues for high & low priorities
as
> follows:
> 
>  
> 
> SERVERHOST              creambo
> 
> ADMIN1                root
> 
> RMCFG[base] TYPE=PBS HOST=creambo EPORT=15004
> 
> SOCKETPROTOCOL=HTTP at RMNMHOST@ NMPORT=12321
> 
> CHARGEPOLICY=DEBITALLWC JOBFAILUREACTION=NONE TIMEOUT=15
> 
> RMPOLLINTERVAL       00:00:02
> 
> SERVERPORT            42559
> 
> SERVERMODE            NORMAL
> 
> LOGFILE               maui.log
> 
> LOGFILEMAXSIZE        10000000
> 
> LOGLEVEL              3
> 
> QUEUETIMEWEIGHT       1
> 
> BACKFILLPOLICY         BESTFIT
> 
> RESERVATIONPOLICY     CURRENTHIGHEST
> 
> NODEALLOCATIONPOLICY  MINRESOURCE
> 
> QOSWEIGHT 1
> 
> CREDWEIGHT 1
> 
> PREEMPTOIONPOLICY REQUEUE
> 
> QOSCFG[hi]  PRIORITY=1000 XFTARGET=100 QFLAGS=PREEMPTOR
> 
> QOSCFG[low] PRIORITY=-100 QFLAGS=PREEMPTEE
> 
> CLASSCFG[long]    QDEF=low
> 
> CLASSCFG[short]   QDEF=hi
> 
>  
> 
>  
> 
> These 2 queues are configured in Qmgr as follows:
> 
>  
> 
> Max open servers: 4
> 
> Qmgr: list queue long
> 
> Queue long
> 
>         queue_type = Execution
> 
>         Priority = 100
> 
>         total_jobs = 4
> 
>         state_count = Transit:0 Queued:4 Held:0 Waiting:0 Running:0
> Exiting:0 
> 
>         max_running = 16
> 
>         enabled = True
> 
>  
> 
> Qmgr: list queue short
> 
> Queue short
> 
>         queue_type = Execution
> 
>         Priority = 1000
> 
>         total_jobs = 0
> 
>         state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0
> Exiting:0 
> 
>         max_running = 16
> 
>         resources_assigned.nodect = 0
> 
>         enabled = True
> 
>  
> 
>  
> 
> Once a high priority job is submitted to a totally busy queue, a low
> priority job is preempted, but there is a problem to restart the
> preempted job once the resource is freed. 
> 
> Such a problem occurs also when using preemption policy of Checkpoint
or
> Suspend.
> 
> Following a Maui log of such a problem:
> 
>  
> 
> 09/28 09:00:28 INFO:     16 feasible tasks found for job 72710:0 in
> partition DEFAULT (1 Needed)
> 
> 09/28 09:00:28 INFO:     tasks located for job 72710:  1 of 1 required
> (1 feasible)
> 
> 09/28 09:00:28 MJobStart(72710)
> 
> 09/28 09:00:28 MJobDistributeTasks(72710,base,NodeList,TaskMap)
> 
> 09/28 09:00:28 MAMAllocJReserve(72710,RIndex,ErrMsg)
> 
> 09/28 09:00:28 MRMJobStart(72710,Msg,SC)
> 
> 09/28 09:00:28 MPBSJobStart(72710,base,Msg,SC)
> 
> 09/28 09:00:28 MPBSJobModify(72710,Resource_List,Resource,wild1)
> 
> 09/28 09:00:28 ERROR:    job '72710' cannot be started: (rc: 15044
> errmsg: 'Resource temporarily unavailable'  hostlist: 'wild1')
> 
> 09/28 09:00:28 MPBSJobModify(72710,Resource_List,Resource,1)
> 
> 09/28 09:00:28 ALERT:    cannot start job 72710 (RM 'base' failed in
> function 'jobstart')
> 
> 09/28 09:00:28 WARNING:  cannot start job '72710' through resource
> manager
> 
> 09/28 09:00:28 ALERT:    job '72710' deferred after 2 failed start
> attempts (API failure on last attempt)
> 
> 09/28 09:00:28 MJobSetHold(72710,16,1:00:00,RMFailure,cannot start job
-
> RM failure, rc: 15044, msg: 'Resource temporarily unavailable')
> 
> 09/28 09:00:28 ALERT:    job '72710' cannot run (deferring job for
3600
> seconds)
> 
> 09/28 09:00:28 MSysRegEvent(JOBDEFER:  defer hold placed on job
'72710'.
> reason: 'RMFailure',0,0,1)
> 
> 09/28 09:00:28 MSysLaunchAction(ASList,1)
> 
> 09/28 09:00:28 ERROR:    cannot start job '72710' in partition DEFAULT
> 
> 09/28 09:00:28 MJobPReserve(72710,DEFAULT,ResCount,ResCountRej)
> 
> 09/28 09:00:28 MJobPReserve(72712,DEFAULT,ResCount,ResCountRej)
> 
>  
> 
>  
> 
> Another job resubmission is failed due to rc=15041.
> 
> I must say that this phenomenon is consistent.
> 
>  
> 
> Is there anybody that has encountered such a problem?
> 
>  
> 


_______________________________________________
mauiusers mailing list
mauiusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/mauiusers




More information about the mauiusers mailing list