[Mauiusers] Problem with Maui when specifying a nodelist with SLURM

Balle, Susanne susanne.balle at hp.com
Fri Jan 14 10:18:40 MST 2005


Hi,

I am running into a problem with Maui.

When submitting jobs with srun using the switch --nodelist=nodename Maui
does not allow me to fill up the whole node (for example to run 2
processes on a 2 processor node). I can only run n-1 processes on an n
processor node. If I do not use the --nodelist switch I can run on n
processes on an n processor node. In the last case things works as
expected. 

The problem is as follows: 
Given a node n10 with 2 processors: 

I would do the following srun command:

"srun -n 2 --nodelist=n10 sleep 10" /* The job gets deferred and never
scheduled.*/

Another way of doing it would be: 

"srun -n 2 sleep 10" and hope that the job gets scheduled onto node n10.
This approach works so I know that Maui can schedule jobs to run on all
the processors in a node.

"srun -n 1 --nodelist=n10 sleep 10" works. Run one 1 processor on node
n10.

>From my experiments it looks like Maui is one off (number of processors
available) when the --nodelist switch has been specified as part of the
srun command.

>From the maui.log below it looks like Maui is ready to schedule the job
on the specified node and then something goes wrong.

I have enclosed the extract of the maui.log that has to do with 
Job 154 ("srun -n 2 --nodelist=xc14n13 sleep 120"). 

In case this is unreadable once the mail is received I have enclosed the
same output in a .txt file.

I increased the LOGLEVEL to 7.

Thanks for any help,

Regards

Susanne

01/14 16:50:19 MWikiGetAttr(job,Name,Status,Attr,Start)
01/14 16:50:19 MJobFind('154',J,0)
01/14 16:50:19 INFO:     job '154'  hash 2857
01/14 16:50:19 MJobCreate(154,JP)
01/14 16:50:19 MJobAddHash(154,1,KIndex)
01/14 16:50:19 INFO:     job slot 1 allocated to job '154'
01/14 16:50:19 MJobFind('154',J,0)
01/14 16:50:19 INFO:     job '154'  hash 2857
01/14 16:50:19 INFO:     job '154' found at hash[2857] 1 '154' (J->Name:
[EMPTY])
01/14 16:50:19 MRMJobPreLoad(J,154,0)
01/14 16:50:19
MWikiJobLoad(154,UPDATETIME=1105739412;STATE=Idle;WCLIMIT=3600;TASKS=2;Q
UEUETIME=1105739412;UNAME=test;GNAME=test;HOSTLIST=xc14n13;PARTITIONMASK
=lsf;NODES=1;RMEM=1;RDISK=1;,J,TaskList,XC14N16)
01/14 16:50:19 MReqCreate(154,SrcRQ,DstRQ,DoCreate)
01/14 16:50:19 INFO:     adding requirement at slot 0
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MWikiUpdateJobAttr(UPDATETIME=1105739412,154)
01/14 16:50:19 MUGetIndex(UPDATETIME=1105739412,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(STATE=Idle,154)
01/14 16:50:19 MUGetIndex(STATE=Idle,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(WCLIMIT=3600,154)
01/14 16:50:19 MUGetIndex(WCLIMIT=3600,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(TASKS=2,154)
01/14 16:50:19 MUGetIndex(TASKS=2,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(QUEUETIME=1105739412,154)
01/14 16:50:19 MUGetIndex(QUEUETIME=1105739412,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(UNAME=test,154)
01/14 16:50:19 MUGetIndex(UNAME=test,ValList,0)
01/14 16:50:19 MUserAdd(UName,UP)
01/14 16:50:19 MWikiUpdateJobAttr(GNAME=test,154)
01/14 16:50:19 MUGetIndex(GNAME=test,ValList,0)
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MWikiUpdateJobAttr(HOSTLIST=xc14n13,154)
01/14 16:50:19 MUGetIndex(HOSTLIST=xc14n13,ValList,0)
01/14 16:50:19 MNodeFind(xc14n13,N)
01/14 16:50:19 MWikiUpdateJobAttr(PARTITIONMASK=lsf,154)
01/14 16:50:19 MUGetIndex(PARTITIONMASK=lsf,ValList,0)
01/14 16:50:19 MUMAFromString(Partition,'lsf',3)
01/14 16:50:19 INFO:     Partition attributes '[lsf]' set
01/14 16:50:19 MWikiUpdateJobAttr(NODES=1,154)
01/14 16:50:19 MUGetIndex(NODES=1,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(RMEM=1,154)
01/14 16:50:19 MUGetIndex(RMEM=1,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(RDISK=1,154)
01/14 16:50:19 MUGetIndex(RDISK=1,ValList,0)
01/14 16:50:19 MJobSetCreds(154,test,test,)
01/14 16:50:19 MUserAdd(UName,UP)
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MJobGetAccount(154,A)
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MRMJobPostLoad(154,TaskList,XC14N16)
01/14 16:50:19 MCPRestore(JOB,154,Optr)
01/14 16:50:19 INFO:     no checkpoint entry for object 'JOB
154 '
01/14 16:50:19 MQOSGetAccess(154,NULL,QAL,QDef)
01/14 16:50:19 INFO:     default QOS for job 154 set to DEFAULT(0)
(P:DEFAULT,U:[NONE],G:[NONE],A:[NONE],C:[NONE])
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MUNLFromTL(NL,TL)
01/14 16:50:19 MJobCheckClassJLimits(154,C,0,Buffer,BufSize)
01/14 16:50:19 MQOSGetAccess(154,NULL,QAL,QDef)
01/14 16:50:19 INFO:     default QOS for job 154 set to DEFAULT(0)
(P:DEFAULT,U:[NONE],G:[NONE],A:[NONE],C:[NONE])
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MJobGetPAL(154,RPAL,PAL,NULL)
01/14 16:50:19 INFO:     job '154' loaded:   2     test     test   3600
Idle   0 1105739412   [NONE] [NONE] [NONE] >=      1 >=      1 [NONE]
1105739412
01/14 16:50:19 INFO:     job '154' size: 0 + 0
01/14 16:50:19 INFO:     1 WIKI jobs detected on RM XC14N16
01/14 16:50:19 INFO:     jobs detected: 1
01/14 16:50:19 MStatClearUsage(node,Active)
01/14 16:50:19 MClusterUpdateNodeState()
01/14 16:50:19 INFO:     node 'xc14n13' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n14' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n15' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n16' C/A/D procs:  4/4/0
01/14 16:50:19 MParUpdate(ALL)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active
0:0
01/14 16:50:19 INFO:     MNode[xc14n13] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n14] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n15] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n16] added to MPar[lsf] (4:4)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active
0:0
01/14 16:50:19 INFO:     jobs in queue
01/14 16:50:19 MResAdjustDRes(NULL,FALSE)
01/14 16:50:19 MQueueSelectAllJobs(Q,HARD,ALL,JIList,DP,Msg)
01/14 16:50:19 MJobGetStartPriority(154,0,Priority,NULL)
01/14 16:50:19 INFO:     job '154' Priority:        1
01/14 16:50:19 INFO:     Cred:      0(00.0)  FS:      0(00.0)  Attr:
0(00.0)  Serv:      0(00.0)  Targ:      0(00.0)  Res:      0(00.0)  Us:
0(00.0)
01/14 16:50:19 INFO:     job '154'  priority:     1.00
01/14 16:50:19 MStatClearUsage([NONE],Active)
01/14 16:50:19 MJobCheckLimits(154,HARD,P,14,Message)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],1,NULL)
01/14 16:50:19 INFO:     job[00] '154' added to master list
01/14 16:50:19 INFO:     total jobs selected (ALL): 1/1 
01/14 16:50:19 INFO:     jobs selected:
[000:   1]
01/14 16:50:19 MQueueSelectAllJobs(Q,SOFT,ALL,JIList,DP,Msg)
01/14 16:50:19 MJobGetStartPriority(154,0,Priority,NULL)
01/14 16:50:19 INFO:     job '154' Priority:        1
01/14 16:50:19 INFO:     Cred:      0(00.0)  FS:      0(00.0)  Attr:
0(00.0)  Serv:      0(00.0)  Targ:      0(00.0)  Res:      0(00.0)  Us:
0(00.0)
01/14 16:50:19 INFO:     job '154'  priority:     1.00
01/14 16:50:19 MStatClearUsage([NONE],Idle)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,14,Message)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],1,NULL)
01/14 16:50:19 INFO:     job[00] '154' added to master list
01/14 16:50:19 INFO:     total jobs selected (ALL): 1/1 
01/14 16:50:19 INFO:     jobs selected:
[000:   1]
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,HARD,5120,4096,2140000000,EVERY,FReason,FALSE
)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,HARD,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,HARD,2,ALL,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,HARD,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 1/1 
01/14 16:50:19 MQueueScheduleSJobs(Q)
01/14 16:50:19 MQueueScheduleRJobs(Q)
01/14 16:50:19 INFO:     checking job 154 in MQueueScheduleRJobs()
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,SOFT,2,ALL,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 1/1 
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,ALL,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 not considered for spanning
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1
[PartitionAccess: 1]
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,lsf,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,SOFT,2,lsf,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition lsf: 1/1 
01/14 16:50:19 MQueueScheduleIJobs(Q,lsf)
01/14 16:50:19 INFO:     checking job '154'
01/14 16:50:19 INFO:     checking job '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 INFO:     checking job 154(1)  state: Idle (ex: Idle)
01/14 16:50:19 MJobSelectMNL(154,lsf,NULL,MNodeList,NodeMap,MaxSpeed,2)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],
INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in
partition lsf (2 Needed)
01/14 16:50:19 MJobGetINL(154,FNL,INL,lsf,NodeCount,TaskCount)
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 INFO:     idle node xc14n13x2 located (D: 0)
01/14 16:50:19 INFO:     idle resources (2 tasks/1 nodes) found with
feasible list specified
01/14 16:50:19 INFO:     adequate idle nodes/tasks located
01/14 16:50:19
MNodeSelectIdleTasks(154,0,SrcNL,IdleMNL,TC,NC,NMap,RCount,RejReason)
01/14 16:50:19
MJobCheckNRes(154,xc14n13,RQ[0],00:00:00,TCAvail,1.000,RIndex,Affinity,F
easCheck)
01/14 16:50:19
MJobCheckNStartTime(154,RQ,xc14n13,00:00:00,TasksAllowed,1.000000,RIndex
,Affinity)
01/14 16:50:19
MJobGetSNRange(154,0,xc14n13,(1 at 00:00:00),1,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for
INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 INFO:     node[0] xc14n13 added to task list (2 tasks : 2
tasks total)
01/14 16:50:19 INFO:     2(0) tasks/1(0) nodes found for job 154 in
MJobSelectMNL
01/14 16:50:19 MJobNLDistribute(154,SrcMNL,DstMNL)
01/14 16:50:19 INFO:     resources found for job 154 tasks: 2+0 of 2
nodes: 1+0 of 1
01/14 16:50:19
MJobAllocMNL(154,MFeasibleList,NodeMap,NULL,MINRESOURCE,1105739419)
01/14 16:50:19 INFO:     using specified hostlist for job 154
01/14 16:50:19 WARNING:  inadequate tasks specified in hostlist for job
154 (1 < 2)
01/14 16:50:19 ERROR:    cannot allocate nodes to job '154' in partition
lsf
01/14 16:50:19 MJobSetAttr(154,SysSMinTime,Value,0,3)
01/14 16:50:19 INFO:     system min start time set on job 154 for
00:00:01
01/14 16:50:19 MJobPReserve(154,lsf,ResCount,ResCountRej)
01/14 16:50:19 MJobReserve(154,Priority)
01/14 16:50:19 MPolicyGetEStartTime(154,ALL,SOFT,Time)
01/14 16:50:19 INFO:     policy start time found for job 154 in 00:00:01
01/14 16:50:19
MJobGetEStartTime(154,NULL,NodeCount,TaskCount,MNodeList,1105739420)
01/14 16:50:19 MParGetTC(lsf,Avl,Cfg,Ded,Req,2140000000)
01/14 16:50:19
MJobGetRange(154,RQ,lsf,00:00:01,GRange,NULL,NodeMap,1,TRange)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],
INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in
partition lsf (2 Needed)
01/14 16:50:19
MJobGetSNRange(154,0,xc14n13,(1 at 00:00:01),256,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for
INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 MRLSFromA(3600,ARL,SRL)
01/14 16:50:19 INFO:     range count: 1
01/14 16:50:19 INFO:     C[00]  S: 1105739420  E: 2139996400  T:   2  N:
1
01/14 16:50:19 MJobSelectFRL(154,G,1,RCount)
01/14 16:50:19 INFO:     start time 00:00:01 found for job 154 in
partition lsf (1105739420)
01/14 16:50:19
MJobGetRange(154,RQ,lsf,00:00:01,GRange,MAvlNodeList,NodeMap,8,NULL)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],
INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in
partition lsf (2 Needed)
01/14 16:50:19
MJobGetSNRange(154,0,xc14n13,(1 at 00:00:01),256,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for
INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 MRLSFromA(3600,ARL,SRL)
01/14 16:50:19 INFO:     node 1 'xc14n13x2' added to nodelist
01/14 16:50:19 INFO:     located resources for 2 tasks (2) in best
partition lsf for job 154 at time 00:00:01
01/14 16:50:19
MJobAllocMNL(154,MFeasibleList,NodeMap,MOutList,MINRESOURCE,1105739420)
01/14 16:50:19 INFO:     using specified hostlist for job 154
01/14 16:50:19 WARNING:  inadequate tasks specified in hostlist for job
154 (1 < 2)
01/14 16:50:19 WARNING:  cannot allocate tasks for job 154 at 00:00:01
01/14 16:50:19 ERROR:    cannot allocate tasks for job 154 at any time
01/14 16:50:19 ALERT:    cannot create new reservation for job 154
(shape[1] 2)
01/14 16:50:19 ALERT:    cannot create new reservation for job 154
01/14 16:50:19 MJobSetHold(154,16,1:00:00,NoResources,cannot create
reservation for job '154' (intital reservation attempt)
)
01/14 16:50:19 ALERT:    job '154' cannot run (deferring job for 3600
seconds)
01/14 16:50:19 WARNING:  cannot reserve priority job '154'
Active Jobs------
------------------
01/14 16:50:19 INFO:     resources available after scheduling: N: 4  P:
10
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],-1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],-1,NULL)
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,HARD,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 rejected (job in non-idle expected
state: 'Deferred')
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1
[EState: 1]
01/14 16:50:19 INFO:     cannot finalize RM cycle (RM 'XC14N16' does not
support function 'cyclefinalize')
01/14 16:50:19
MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 rejected (job in non-idle expected
state: 'Deferred')
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1
[EState: 1]
01/14 16:50:19 MSchedUpdateStats()
01/14 16:50:19 INFO:     iteration: 8074   scheduling time:  0.005
seconds
01/14 16:50:19 MResUpdateStats()
01/14 16:50:19 INFO:     current util[8074]:  0/4 (0.00%)  PH: 0.12%
active jobs: 0 of 2 (completed: 195)
01/14 16:50:19 MQueueCheckStatus()
01/14 16:50:19 INFO:     checking purge criteria for job '154'
01/14 16:50:19 MNodeCheckStatus()
01/14 16:50:19 INFO:     checking node 'xc14n13'
01/14 16:50:19 INFO:     checking node 'xc14n14'
01/14 16:50:19 INFO:     checking node 'xc14n15'
01/14 16:50:19 INFO:     checking node 'xc14n16'
01/14 16:50:19 MSysCheck()
01/14 16:50:19 MLimitEnforceAll(ALL)
01/14 16:50:19 MUClearChild(PID)
01/14 16:50:19 MParUpdate(ALL)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active
0:0
01/14 16:50:19 INFO:     MNode[xc14n13] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n14] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n15] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n16] added to MPar[lsf] (4:4)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active
0:0
01/14 16:50:19 MResCheckStatus(NULL)
01/14 16:50:19 INFO:     scheduling complete.  sleeping 10 seconds
-------------- next part --------------
01/14 16:50:19 MWikiGetAttr(job,Name,Status,Attr,Start)
01/14 16:50:19 MJobFind('154',J,0)
01/14 16:50:19 INFO:     job '154'  hash 2857
01/14 16:50:19 MJobCreate(154,JP)
01/14 16:50:19 MJobAddHash(154,1,KIndex)
01/14 16:50:19 INFO:     job slot 1 allocated to job '154'
01/14 16:50:19 MJobFind('154',J,0)
01/14 16:50:19 INFO:     job '154'  hash 2857
01/14 16:50:19 INFO:     job '154' found at hash[2857] 1 '154' (J->Name: [EMPTY])
01/14 16:50:19 MRMJobPreLoad(J,154,0)
01/14 16:50:19 MWikiJobLoad(154,UPDATETIME=1105739412;STATE=Idle;WCLIMIT=3600;TASKS=2;QUEUETIME=1105739412;UNAME=test;GNAME=test;HOSTLIST=xc14n13;PARTITIONMASK=lsf;NODES=1;RMEM=1;RDISK=1;,J,TaskList,XC14N16)
01/14 16:50:19 MReqCreate(154,SrcRQ,DstRQ,DoCreate)
01/14 16:50:19 INFO:     adding requirement at slot 0
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MWikiUpdateJobAttr(UPDATETIME=1105739412,154)
01/14 16:50:19 MUGetIndex(UPDATETIME=1105739412,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(STATE=Idle,154)
01/14 16:50:19 MUGetIndex(STATE=Idle,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(WCLIMIT=3600,154)
01/14 16:50:19 MUGetIndex(WCLIMIT=3600,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(TASKS=2,154)
01/14 16:50:19 MUGetIndex(TASKS=2,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(QUEUETIME=1105739412,154)
01/14 16:50:19 MUGetIndex(QUEUETIME=1105739412,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(UNAME=test,154)
01/14 16:50:19 MUGetIndex(UNAME=test,ValList,0)
01/14 16:50:19 MUserAdd(UName,UP)
01/14 16:50:19 MWikiUpdateJobAttr(GNAME=test,154)
01/14 16:50:19 MUGetIndex(GNAME=test,ValList,0)
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MWikiUpdateJobAttr(HOSTLIST=xc14n13,154)
01/14 16:50:19 MUGetIndex(HOSTLIST=xc14n13,ValList,0)
01/14 16:50:19 MNodeFind(xc14n13,N)
01/14 16:50:19 MWikiUpdateJobAttr(PARTITIONMASK=lsf,154)
01/14 16:50:19 MUGetIndex(PARTITIONMASK=lsf,ValList,0)
01/14 16:50:19 MUMAFromString(Partition,'lsf',3)
01/14 16:50:19 INFO:     Partition attributes '[lsf]' set
01/14 16:50:19 MWikiUpdateJobAttr(NODES=1,154)
01/14 16:50:19 MUGetIndex(NODES=1,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(RMEM=1,154)
01/14 16:50:19 MUGetIndex(RMEM=1,ValList,0)
01/14 16:50:19 MWikiUpdateJobAttr(RDISK=1,154)
01/14 16:50:19 MUGetIndex(RDISK=1,ValList,0)
01/14 16:50:19 MJobSetCreds(154,test,test,)
01/14 16:50:19 MUserAdd(UName,UP)
01/14 16:50:19 MGroupAdd(GName,GP)
01/14 16:50:19 MJobGetAccount(154,A)
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MRMJobPostLoad(154,TaskList,XC14N16)
01/14 16:50:19 MCPRestore(JOB,154,Optr)
01/14 16:50:19 INFO:     no checkpoint entry for object 'JOB                        154 '
01/14 16:50:19 MQOSGetAccess(154,NULL,QAL,QDef)
01/14 16:50:19 INFO:     default QOS for job 154 set to DEFAULT(0) (P:DEFAULT,U:[NONE],G:[NONE],A:[NONE],C:[NONE])
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MUNLFromTL(NL,TL)
01/14 16:50:19 MJobCheckClassJLimits(154,C,0,Buffer,BufSize)
01/14 16:50:19 MQOSGetAccess(154,NULL,QAL,QDef)
01/14 16:50:19 INFO:     default QOS for job 154 set to DEFAULT(0) (P:DEFAULT,U:[NONE],G:[NONE],A:[NONE],C:[NONE])
01/14 16:50:19 INFO:     job flags for job 154: 40
01/14 16:50:19 MJobSetAttr(154,GAttr,Value,1,5)
01/14 16:50:19 MJobGetPAL(154,RPAL,PAL,NULL)
01/14 16:50:19 INFO:     job '154' loaded:   2     test     test   3600       Idle   0 1105739412   [NONE] [NONE] [NONE] >=      1 >=      1 [NONE] 1105739412
01/14 16:50:19 INFO:     job '154' size: 0 + 0
01/14 16:50:19 INFO:     1 WIKI jobs detected on RM XC14N16
01/14 16:50:19 INFO:     jobs detected: 1
01/14 16:50:19 MStatClearUsage(node,Active)
01/14 16:50:19 MClusterUpdateNodeState()
01/14 16:50:19 INFO:     node 'xc14n13' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n14' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n15' C/A/D procs:  2/2/0
01/14 16:50:19 INFO:     node 'xc14n16' C/A/D procs:  4/4/0
01/14 16:50:19 MParUpdate(ALL)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active 0:0
01/14 16:50:19 INFO:     MNode[xc14n13] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n14] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n15] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n16] added to MPar[lsf] (4:4)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active 0:0
01/14 16:50:19 INFO:     jobs in queue
01/14 16:50:19 MResAdjustDRes(NULL,FALSE)
01/14 16:50:19 MQueueSelectAllJobs(Q,HARD,ALL,JIList,DP,Msg)
01/14 16:50:19 MJobGetStartPriority(154,0,Priority,NULL)
01/14 16:50:19 INFO:     job '154' Priority:        1
01/14 16:50:19 INFO:     Cred:      0(00.0)  FS:      0(00.0)  Attr:      0(00.0)  Serv:      0(00.0)  Targ:      0(00.0)  Res:      0(00.0)  Us:      0(00.0)
01/14 16:50:19 INFO:     job '154'  priority:     1.00
01/14 16:50:19 MStatClearUsage([NONE],Active)
01/14 16:50:19 MJobCheckLimits(154,HARD,P,14,Message)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],1,NULL)
01/14 16:50:19 INFO:     job[00] '154' added to master list
01/14 16:50:19 INFO:     total jobs selected (ALL): 1/1 
01/14 16:50:19 INFO:     jobs selected:
[000:   1]
01/14 16:50:19 MQueueSelectAllJobs(Q,SOFT,ALL,JIList,DP,Msg)
01/14 16:50:19 MJobGetStartPriority(154,0,Priority,NULL)
01/14 16:50:19 INFO:     job '154' Priority:        1
01/14 16:50:19 INFO:     Cred:      0(00.0)  FS:      0(00.0)  Attr:      0(00.0)  Serv:      0(00.0)  Targ:      0(00.0)  Res:      0(00.0)  Us:      0(00.0)
01/14 16:50:19 INFO:     job '154'  priority:     1.00
01/14 16:50:19 MStatClearUsage([NONE],Idle)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,14,Message)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],1,NULL)
01/14 16:50:19 INFO:     job[00] '154' added to master list
01/14 16:50:19 INFO:     total jobs selected (ALL): 1/1 
01/14 16:50:19 INFO:     jobs selected:
[000:   1]
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,HARD,5120,4096,2140000000,EVERY,FReason,FALSE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,HARD,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,HARD,2,ALL,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,HARD,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 1/1 
01/14 16:50:19 MQueueScheduleSJobs(Q)
01/14 16:50:19 MQueueScheduleRJobs(Q)
01/14 16:50:19 INFO:     checking job 154 in MQueueScheduleRJobs()
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,SOFT,2,ALL,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 1/1 
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,ALL,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 not considered for spanning
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1 [PartitionAccess: 1]
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,lsf,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,8,Message)
01/14 16:50:19 MJobCheckPolicies(154,SOFT,2,lsf,RIndex,NULL,2140000000)
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 MLocalCheckFairnessPolicy(154,1105739419,Message)
01/14 16:50:19 INFO:     job '154' added to queue at slot 0
01/14 16:50:19 INFO:     total jobs selected in partition lsf: 1/1 
01/14 16:50:19 MQueueScheduleIJobs(Q,lsf)
01/14 16:50:19 INFO:     checking job '154'
01/14 16:50:19 INFO:     checking job '154'
01/14 16:50:19 MJobCheckLimits(154,SOFT,P,2,Message)
01/14 16:50:19 INFO:     checking job 154(1)  state: Idle (ex: Idle)
01/14 16:50:19 MJobSelectMNL(154,lsf,NULL,MNodeList,NodeMap,MaxSpeed,2)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],  INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in partition lsf (2 Needed)
01/14 16:50:19 MJobGetINL(154,FNL,INL,lsf,NodeCount,TaskCount)
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 INFO:     idle node xc14n13x2 located (D: 0)
01/14 16:50:19 INFO:     idle resources (2 tasks/1 nodes) found with feasible list specified
01/14 16:50:19 INFO:     adequate idle nodes/tasks located
01/14 16:50:19 MNodeSelectIdleTasks(154,0,SrcNL,IdleMNL,TC,NC,NMap,RCount,RejReason)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],00:00:00,TCAvail,1.000,RIndex,Affinity,FeasCheck)
01/14 16:50:19 MJobCheckNStartTime(154,RQ,xc14n13,00:00:00,TasksAllowed,1.000000,RIndex,Affinity)
01/14 16:50:19 MJobGetSNRange(154,0,xc14n13,(1 at 00:00:00),1,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for   INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 INFO:     node[0] xc14n13 added to task list (2 tasks : 2 tasks total)
01/14 16:50:19 INFO:     2(0) tasks/1(0) nodes found for job 154 in MJobSelectMNL
01/14 16:50:19 MJobNLDistribute(154,SrcMNL,DstMNL)
01/14 16:50:19 INFO:     resources found for job 154 tasks: 2+0 of 2  nodes: 1+0 of 1
01/14 16:50:19 MJobAllocMNL(154,MFeasibleList,NodeMap,NULL,MINRESOURCE,1105739419)
01/14 16:50:19 INFO:     using specified hostlist for job 154
01/14 16:50:19 WARNING:  inadequate tasks specified in hostlist for job 154 (1 < 2)
01/14 16:50:19 ERROR:    cannot allocate nodes to job '154' in partition lsf
01/14 16:50:19 MJobSetAttr(154,SysSMinTime,Value,0,3)
01/14 16:50:19 INFO:     system min start time set on job 154 for 00:00:01
01/14 16:50:19 MJobPReserve(154,lsf,ResCount,ResCountRej)
01/14 16:50:19 MJobReserve(154,Priority)
01/14 16:50:19 MPolicyGetEStartTime(154,ALL,SOFT,Time)
01/14 16:50:19 INFO:     policy start time found for job 154 in 00:00:01
01/14 16:50:19 MJobGetEStartTime(154,NULL,NodeCount,TaskCount,MNodeList,1105739420)
01/14 16:50:19 MParGetTC(lsf,Avl,Cfg,Ded,Req,2140000000)
01/14 16:50:19 MJobGetRange(154,RQ,lsf,00:00:01,GRange,NULL,NodeMap,1,TRange)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],  INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in partition lsf (2 Needed)
01/14 16:50:19 MJobGetSNRange(154,0,xc14n13,(1 at 00:00:01),256,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for   INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 MRLSFromA(3600,ARL,SRL)
01/14 16:50:19 INFO:     range count: 1
01/14 16:50:19 INFO:     C[00]  S: 1105739420  E: 2139996400  T:   2  N: 1
01/14 16:50:19 MJobSelectFRL(154,G,1,RCount)
01/14 16:50:19 INFO:     start time 00:00:01 found for job 154 in partition lsf (1105739420)
01/14 16:50:19 MJobGetRange(154,RQ,lsf,00:00:01,GRange,MAvlNodeList,NodeMap,8,NULL)
01/14 16:50:19 MReqGetFNL(154,0,lsf,NULL,DstNL,NC,TC,2140000000,0)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,NULL)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 MNodeCheckPolicies(154,xc14n13,2)
01/14 16:50:19 MJobCheckNRes(154,xc14n13,RQ[0],  INFINITY,TCAvail,1.000,RIndex,NULL,FeasCheck)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n13,RIndex)
01/14 16:50:19 INFO:     node in requested hostlist
01/14 16:50:19 INFO:     node xc14n13 added to feasible list (2 tasks)
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n14,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n15,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 MReqCheckResourceMatch(154,0,xc14n16,NULL)
01/14 16:50:19 INFO:     node is not in specified hostlist
01/14 16:50:19 INFO:     2 feasible tasks found for job 154:0 in partition lsf (2 Needed)
01/14 16:50:19 MJobGetSNRange(154,0,xc14n13,(1 at 00:00:01),256,Affinity,Type,ARange,BRes)
01/14 16:50:19 MRECheck(xc14n13,MJobGetSNRange-Start,FORCE)
01/14 16:50:19 INFO:     node xc14n13 supports 2 tasks of job 154:0 for   INFINITY of 1:00:00 (no reservation)
01/14 16:50:19 MRLSFromA(3600,ARL,SRL)
01/14 16:50:19 INFO:     node 1 'xc14n13x2' added to nodelist
01/14 16:50:19 INFO:     located resources for 2 tasks (2) in best partition lsf for job 154 at time 00:00:01
01/14 16:50:19 MJobAllocMNL(154,MFeasibleList,NodeMap,MOutList,MINRESOURCE,1105739420)
01/14 16:50:19 INFO:     using specified hostlist for job 154
01/14 16:50:19 WARNING:  inadequate tasks specified in hostlist for job 154 (1 < 2)
01/14 16:50:19 WARNING:  cannot allocate tasks for job 154 at 00:00:01
01/14 16:50:19 ERROR:    cannot allocate tasks for job 154 at any time
01/14 16:50:19 ALERT:    cannot create new reservation for job 154 (shape[1] 2)
01/14 16:50:19 ALERT:    cannot create new reservation for job 154
01/14 16:50:19 MJobSetHold(154,16,1:00:00,NoResources,cannot create reservation for job '154' (intital reservation attempt)
)
01/14 16:50:19 ALERT:    job '154' cannot run (deferring job for 3600 seconds)
01/14 16:50:19 WARNING:  cannot reserve priority job '154'
Active Jobs------
------------------
01/14 16:50:19 INFO:     resources available after scheduling: N: 4  P: 10
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,PU,[ALL],-1,NULL)
01/14 16:50:19 MPolicyAdjustUsage(NULL,154,NULL,idle,NULL,[ALL],-1,NULL)
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,HARD,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 rejected (job in non-idle expected state: 'Deferred')
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1 [EState: 1]
01/14 16:50:19 INFO:     cannot finalize RM cycle (RM 'XC14N16' does not support function 'cyclefinalize')
01/14 16:50:19 MQueueSelectJobs(SrcQ,DstQ,SOFT,5120,4096,2140000000,EVERY,FReason,TRUE)
01/14 16:50:19 MLocalCheckFairnessPolicy(NULL,1105739419,Message)
01/14 16:50:19 INFO:     checking job[0] '154'
01/14 16:50:19 INFO:     job 154 rejected (job in non-idle expected state: 'Deferred')
01/14 16:50:19 INFO:     total jobs selected in partition ALL: 0/1 [EState: 1]
01/14 16:50:19 MSchedUpdateStats()
01/14 16:50:19 INFO:     iteration: 8074   scheduling time:  0.005 seconds
01/14 16:50:19 MResUpdateStats()
01/14 16:50:19 INFO:     current util[8074]:  0/4 (0.00%)  PH: 0.12%  active jobs: 0 of 2 (completed: 195)
01/14 16:50:19 MQueueCheckStatus()
01/14 16:50:19 INFO:     checking purge criteria for job '154'
01/14 16:50:19 MNodeCheckStatus()
01/14 16:50:19 INFO:     checking node 'xc14n13'
01/14 16:50:19 INFO:     checking node 'xc14n14'
01/14 16:50:19 INFO:     checking node 'xc14n15'
01/14 16:50:19 INFO:     checking node 'xc14n16'
01/14 16:50:19 MSysCheck()
01/14 16:50:19 MLimitEnforceAll(ALL)
01/14 16:50:19 MUClearChild(PID)
01/14 16:50:19 MParUpdate(ALL)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active 0:0
01/14 16:50:19 INFO:     MNode[xc14n13] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n14] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n15] added to MPar[lsf] (2:2)
01/14 16:50:19 INFO:     MNode[xc14n16] added to MPar[lsf] (4:4)
01/14 16:50:19 INFO:     P[ALL]:  Total 4:10  Up 4:10  Idle 4:10  Active 0:0
01/14 16:50:19 MResCheckStatus(NULL)
01/14 16:50:19 INFO:     scheduling complete.  sleeping 10 seconds


More information about the mauiusers mailing list