[torqueusers] Fwd: ncpus anyone?

David Singleton David.Singleton at anu.edu.au
Sat Mar 13 16:58:53 MST 2010

On 03/13/2010 06:22 PM, Chris Samuel wrote:
> On Wed, 3 Mar 2010 06:12:14 am David Beer wrote:
>> Another enhancement that we're considering is locking memory close to the
>> CPUs (where relevant). I know that on some systems (NUMA, for example)
>> this is necessary in order to run jobs efficiently.
> Very much agree - can I suggest that you look at the hwloc library being
> developed by the Open-MPI developers to provide a way of learning the
> architecture of systems as part of this ?
> Also you may well want to look at supporting control groups (cgroups) in
> addition to cpusets as that provides some really handy extra features like
> being able to limit the memory usage of a cgroup, good resource usage
> measurement, etc.

Hi Chris

I was interested in cgroups as well but this recent post to the SLURM list
is a bit off-putting.


-------- Original Message --------
Subject: [slurm-dev] proctrack/cgroup beta version warnings
Date: Tue, 9 Mar 2010 23:07:59 +0100
From: matthieu hautreux <matthieu.hautreux at gmail.com>
Reply-To: slurm-dev at lists.llnl.gov
To: slurm-dev <slurm-dev at lists.llnl.gov>


recently, I sent a patch to add support of a beta plugin for proctrack in
slurm using cgroup.
Additional tests and usages showed us that using cgroup memory subsystem on
SMP nodes has a huge overhead that explodes with the number of cores used
simultaneously. This seems to be due to locking problems in kernel cgroup
memory subsystem structures involved in the memory accounting. When a lot of
pages are allocated/freed simultaneously on multiple cores, the lock of
internal kernel structures seems to become a big bottleneck (timex20 on 32

I strongly suggest not to use this beta version as it is based on this
problematic memory subsystem for now. I plan to write a new version that
will use an other cgroup subsystem (freezer) to track process instead of the
memory one. The memory subsystem will probably be available but not
configured by default. I hope that this bottleneck will be
reduced/eliminated in a future version of kernel memory cgroup support, we
will try to open a ticket ASAP if our feelings are confirmed by kernel
sources study. Unfortunately, this new version will not be available before
some weeks.


More information about the torqueusers mailing list