[torquedev] Feedback on per vnode cpusets

[Gareth.Williams at csiro.au]

> -----Original Message-----
> From: torquedev-bounces at supercluster.org 
> [mailto:torquedev-bounces at supercluster.org] On Behalf Of Chris Samuel
> Sent: Saturday, 24 May 2008 8:11 PM
> To: torquedev
> Subject: [torquedev] Feedback on per vnode cpusets
> 
> Hi all,
> 
> I'm currently looking at the cpuset code in trunk and
> wondering what to do about the per-vnode cpusets.
> 
> It's a trivial change to pbs_mom put the tm_spawn children
> into the job cpuset, so the question is rather how to
> handle the differences.
> 
> In either of the cases where the per-vnode cpusets remain
> I would strongly suggest that they default to off (so as
> not to surprise unsuspecting sites).
> 
> So, the options I can see are:
> 
> 1) per-vnode cpusets are a compile time option
> 
> Requires some autoconf tweaks and a #ifdef and if
> a site changes their mind its a recompile, reinstall
> and restart pbs_mom cycle.
> 
> 2) per-vnode cpusets are a pbs_mom.conf option
> 
> Plagiarise, er, be inspired by the enablemomrestart
> code and create a new global variable.  If a site changes
> its mind it changes the config file and reloads/restarts
> pbs_mom.
> 
> 3) per-vnode cpusets go altogether
> 
> Easiest to do, but removes the ability for sites using
> MPICH-1 based MPI's like MPICH-GM, MVAPICH1, etc, from
> being able to lock individual tasks to cores.
> 
> 
> Thoughts ?
> 
> cheers!
> Chris
> -- 
> Christopher Samuel - (03) 9925 4751 - Systems Manager
>  The Victorian Partnership for Advanced Computing
>  P.O. Box 201, Carlton South, VIC 3053, Australia
> VPAC is a not-for-profit Registered Research Agency
> _______________________________________________
> torquedev mailing list
> torquedev at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torquedev
> 

Hi All,

A 4th option is to choose whether or not to use the vnode cpusets on a
per-job or per-process basis.  I think tm_spawn will be the principle
(only!) method of distributing processes to nodes/cpustes and it could
honor an environment variable or a property associated with the job.

This would be in combination with 1), but I would tend to favour having
the vnode cpusets available but not usually using them.  

-- Gareth

Note. If (the vnode cpusets were not available but) the job cpuset was
user writeable, the user could construct their own sub-cpusets to
contain processes, but it would be hard work.