[torqueusers] Torque free/job-exclusive confusion

garrick at speculation.org garrick at speculation.org
Wed Jun 14 19:31:24 MDT 2006

On Wed, Jun 14, 2006 at 01:03:05PM +0200, Lennart Karlsson alleged:
> Torque, versions 2.1.1-snap.200605191740 and 2.1.0p0-snap.200603212346,
> does something that confuses me and also seems to confuse Maui and Moab,
> making their scheduling more difficult than necessary:

How is maui/moab getting confused?

> It sometimes sets the state "free" on busy nodes.
> I see this behaviour on both one-processor and two-processor (where both
> processors are used) nodes.
> E.g. do I get this output from 'pbsnodes -a n1':
> n1
>      state = free
>      np = 1
>      ntype = cluster
>      jobs = 0/122033.green
>      status = opsys=linux,uname=Linux n1 2.4.21-27.0.2.EL-nird1 #1 Wed Feb 9 
> 15:58:46 CET 2005 i686,sessions=4127,nsessions=1,nusers=1,
> idletime=19344652,totmem=4106380kb,availmem=4017800kb,
> physmem=2058104kb,ncpus=1,loadave=0.99,netload=3709266365,
> state=free,jobs=122033.green,
> rectime=1150279971

The node is not "busy".  You only get a busy state when MOM reports
itself as busy, which is based on the load average and the $ideal_load
and $max_load config parameters.

> If I restart the pbs_server I get the (in my eyes) more sane:
> n1
>      state = job-exclusive
>      np = 1
>      ntype = cluster
>      jobs = 0/122033.green
>      status = opsys=linux,uname=Linux n1 2.4.21-27.0.2.EL-nird1 #1 Wed Feb 9 
> 15:58:46 CET 2005 i686,sessions=4127,nsessions=1,nusers=1,idletime=19346814,tot
> mem=4106380kb,availmem=4017864kb,physmem=2058104kb,ncpus=1,loadave=0.99,netload
> =3709431732,state=free,jobs=122033.green,rectime=1150282133

Hrm, odd.  It shouldn't be different after a restart.

I haven't played with single proc nodes much, so that will require some
testing on my part.

Is that single-node or multi-node job?

> I do not really know when and why the node is declared "free". The job
> 122033 had run for a few hours before this happened, so it does not seem
> to be a lost communication between pbs_mom and pbs_server.
> Is there a fix to this problem? Or do someone of you have a nice idea about
> how to find where the information mismatch is created? I assume that the
> state information is set by the pbs_server? Or is it set by the scheduler
> or the pbs_mom?

The state is set by pbs_server, but is based on several factors.
 - job-exclusive, which seems entirely useless, happens when all
   CPUs of a node go to the same job.
 - busy happens when MOM reports itself as busy.
 - down means pbs_server isn't talking to MOM
 - unknown means pbs_server hasn't yet established a connection and
   doesn't know what else to report.
 - offline prevents new jobs and is explicitly set by an admin
 - free is the absense of all other states.

More information about the torqueusers mailing list