[Mauiusers] Nodes go to "State: Drained" without being asked to do so

[Itay M]

Hello,
We have a Torque/PBS/MAUI system with 30 compute nodes.
During the last few days, we expirience a strange phenomena : certain
compute nodes go to 'State: Drained' , while the cluster admins didn't ask
these nodes to go to drained mode.

We found that using the following command says:


===============================
[root at cluster sbin]$ checknode node23
checking node bioc23.tau.ac.il
State:   Drained  (in current state for 1:02:30:43)
===============================

We can set the node back to Idle by using

===============================
mnodectl host=node23 modify state=Idle
===============================

This works fine. But why do nodes go drained state without being asked for,
and how can we troubleshoot or prevent this from happenins?

Thanks,
Itay.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/mauiusers/attachments/20090105/3f0b27c5/attachment.html