[torqueusers] Removing processes after a job is killed
jbernstein at penguincomputing.com
Wed Jul 2 12:12:34 MDT 2008
Prakash Velayutham wrote:
> I am sure you must have heard of mpiexec for Torque-based Task
> Management. mpiexec (available from www.osc.edu/~pw/mpiexec/index.php)
> basically does the cleanup for you when you do qdel or something like that.
You should definitely be using mpiexec when running a MPICH or MVAPICH
job under TORQUE. mpiexec uses the tm interface to spawn processes on
remote nodes, rather then using something like SSH. The benefit of using
the tm interface is two fold. First, the issue you describe will go away
because suddenly the sister mom's will know which processes to kill when
a qdel or other kill signal is received. Second, utilizations rates will
be tracked across ALL of the nodes in the job rather then just on the
mom superior. This is especially important when using something like
Moab or other statistics package to track utilization or do chargeback.
Without mpiexec, users will be getting with cycles they haven't paid for.
> On Jul 2, 2008, at 1:42 PM, Craig Macdonald wrote:
>> Firstly, limit each node to one job per user. Then you can use a kill
>> in the epilogue. See below for cutdown example
>> ps -U $userid -o pid --no-heading | xargs -r kill -KILL
>> David Sheen wrote:
>>> The parallel programming environments we use (e.g. MPICH) use SSH to
>>> create processes on the sister nodes. If these jobs fail (are
>>> deleted, the mother node crashes, etc), the spawned processes remain
>>> on the sisters and eventually someone has to go and clean them out.
>>> Is there any way to use epilogue scripts to keep track of these
>>> processes and make sure they get killed properly if they need to be?
>>> torqueusers mailing list
>>> torqueusers at supercluster.org
>> torqueusers mailing list
>> torqueusers at supercluster.org
> Prakash Velayutham
> Programmer / Analyst
> Cincinnati Children's Hospital Medical Center
> torqueusers mailing list
> torqueusers at supercluster.org
More information about the torqueusers