[torquedev] [PATCH] Change pbs_mom to set RLIMIT_AS instead of RLIMIT_DATA for mem/pmem limits.

[Åke Sandgren]

On Sun, 2009-01-11 at 14:07 +1100, David Singleton wrote:
> Chris Samuel wrote:
> > ----- "David Singleton" <David.Singleton at anu.edu.au> wrote:
> > 
> > Hi Dave!
> > 
> >> Hi Chris,
> >>
> >> I have to agree with Ake:
> >>
> >>   * mem/pmem is for limiting RSS/physical memory use
> >>   * vmem/pvmem is for limiting virtual memory/address ranges
> > 
> > But that's not what pbs_mom is designed to do, 
> 
> Hmmm, well it does in OpenPBS.  Looks like Torque broke this at some
> stage.   But the Torque doco still describes how it should be:

It's not really broken in Torque. It still sets _RSS for pmem but it
also sets DATA and STACK for some reason.

> mem    Maximum amount of physical memory used by the job.
> pmem   Maximum amount of physical memory used by any single process of the job.
> pvmem  Maximum amount of virtual memory used by any single process in the job.
> vmem   Maximum amount of virtual memory used by all concurrent processes in the job.
> 
> How did RLIMIT_DATA get into limiting "mem"?

Not sure. I don't have anything older then 1.2.0p6 of torque available
and that already contained my patch for the change to use _AS for pvmem.
I don't have my original patch around any more.

In the current situation with the kernel not enforcing _RSS one should
use pvmem and not pmem as the limiting factor for memory.

There might be a way to solve this if the kernel starts using memorysets
(like cpusets) but i haven't seen any real plans yet.

To make things more complicated there are situations where one would
like to have different pvmem limits on different nodes in a job.
i.e. nodes=1+2:ppn=8 with pvmem=16g+16:2g meaning 1 node with 16g and 2
nodes with 8 cores each with 2g.

So this resource limit setup really needs a good overhaul.

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake at hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se